You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Vladimir Ozerov (JIRA)" <ji...@apache.org> on 2018/10/26 07:25:00 UTC
[jira] [Assigned] (IGNITE-10010) Node halted if second node was
stopped, then cache destroyed, then second node returned
[ https://issues.apache.org/jira/browse/IGNITE-10010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vladimir Ozerov reassigned IGNITE-10010:
----------------------------------------
Assignee: Alexey Goncharuk
> Node halted if second node was stopped, then cache destroyed, then second node returned
> ---------------------------------------------------------------------------------------
>
> Key: IGNITE-10010
> URL: https://issues.apache.org/jira/browse/IGNITE-10010
> Project: Ignite
> Issue Type: Bug
> Components: mvcc
> Affects Versions: 2.7
> Reporter: Sergey Kozlov
> Assignee: Alexey Goncharuk
> Priority: Blocker
> Fix For: 2.7
>
> Attachments: PersistenceNodeRestartAfterCacheDropSelfTest.java
>
>
> 1. Start 2 nodes with PDS
> 2. Activate cluster
> 3. Connect sqlline.
> 4. Create table {{create table t1(a int, b varchar, primary key(a)) with "ATOMICITY=TRANSACTIONAL_SNAPSHOT,backups=1";}}
> 5. Stop node 1
> 6. Drop table {{drop table t1;}}
> 7. Start node 1
> 8. Node 2 stopped by handler:
> {noformat}
> c:\Work\apache-ignite-2.7.0-SNAPSHOT-bin>bin\ignite.bat server.xml -v -J-DID=1
> Ignite Command Line Startup, ver. 2.7.0-SNAPSHOT#19700101-sha1:DEV
> 2018 Copyright(C) Apache Software Foundation
> [18:04:22,745][INFO][main][IgniteKernal]
> >>> __________ ________________
> >>> / _/ ___/ |/ / _/_ __/ __/
> >>> _/ // (7 7 // / / / / _/
> >>> /___/\___/_/|_/___/ /_/ /___/
> >>>
> >>> ver. 2.7.0-SNAPSHOT#19700101-sha1:DEV
> >>> 2018 Copyright(C) Apache Software Foundation
> >>>
> >>> Ignite documentation: http://ignite.apache.org
> [18:04:22,745][INFO][main][IgniteKernal] Config URL: file:/c:/Work/apache-ignite-2.7.0-SNAPSHOT-bin/server.xml
> [18:04:22,760][INFO][main][IgniteKernal] IgniteConfiguration [igniteInstanceName=null, pubPoolSize=8, svcPoolSize=8, cal
> lbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=8, dataStreamerPoolSize=8, utilityCacheP
> oolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=c:\Work\apache-ignite-2.7.0-SNAPSHO
> T-bin, igniteWorkDir=c:\Work\apache-ignite-2.7.0-SNAPSHOT-bin\work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94
> fa3e, nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabl
> ed=false, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpT
> ime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10,
> reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null], segPlc=ST
> OP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSp
> i [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@22ff4249,
> enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=47100, locPortRange=1
> 00, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, r
> econCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, us
> ePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, una
> ckedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRs
> lvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@2d1ef81a[Count = 1], stopping=false], evtSpi=org.apache.ignit
> e.spi.eventstorage.NoopEventStorageSpi@4c402120, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSp
> i=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@815b41f, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encry
> ption.noop.NoopEncryptionSpi@5542c4ed, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSe
> rEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExch
> ange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=fa
> lse], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.
> 1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailu
> reDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=ConnectorConfiguration [jettyPath=null, hos
> t=null, port=11211, noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=600000, idleQry
> CurCheckFreq=60000, sndQueueLimit=0, selectorCnt=4, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, sslCtxFacto
> ry=null, sslFactory=null, portRange=100, threadPoolSize=8, msgInterceptor=null], odbcCfg=null, warmupClos=null, atomicCf
> g=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, ss
> lCtxFactory=null, platformCfg=null, binaryCfg=BinaryConfiguration [idMapper=null, nameMapper=null, serializer=null, comp
> actFooter=true], memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=52428800, sysRegionMaxSize=
> 104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=104857600, initSize=524
> 28800, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, me
> tricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=true, checkpointPageBufSize=0], dataRegions=D
> ataRegionConfiguration[] [DataRegionConfiguration [name=dataRegion1, maxSize=209715200, initSize=104857600, swapPath=nul
> l, pageEvictionMode=RANDOM_LRU, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalC
> ount=5, metricsRateTimeInterval=60000, persistenceEnabled=true, checkpointPageBufSize=0], DataRegionConfiguration [name=
> dataRegion2, maxSize=104857600, initSize=52428800, swapPath=null, pageEvictionMode=RANDOM_LRU, evictionThreshold=0.9, em
> ptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled
> =true, checkpointPageBufSize=0]], storagePath=null, checkpointFreq=1000, lockWaitTime=10000, checkpointThreads=4, checkp
> ointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPat
> h=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFl
> ushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.i
> gnite.internal.processors.cache.persistence.file.AsyncFileIOFactory@3cd3e762, metricsSubIntervalCnt=5, metricsRateTimeIn
> terval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionL
> evel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=nu
> ll, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, t
> cpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliE
> nabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCn
> t=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=null, commFailureRslvr=null]
> [18:04:22,760][INFO][main][IgniteKernal] Daemon mode: off
> [18:04:22,760][INFO][main][IgniteKernal] OS: Windows 10 10.0 amd64
> [18:04:22,760][INFO][main][IgniteKernal] OS user: gridgain
> [18:04:22,760][INFO][main][IgniteKernal] PID: 836
> [18:04:22,760][INFO][main][IgniteKernal] Language runtime: Java Platform API Specification ver. 1.8
> [18:04:22,760][INFO][main][IgniteKernal] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporatio
> n Java HotSpot(TM) 64-Bit Server VM 25.181-b13
> [18:04:22,776][INFO][main][IgniteKernal] VM total memory: 0.96GB
> [18:04:22,776][INFO][main][IgniteKernal] Remote Management [restart: on, REST: on, JMX (remote: on, port: 49148, auth: o
> ff, ssl: off)]
> [18:04:22,776][INFO][main][IgniteKernal] Logger: JavaLogger [quiet=false, config=null]
> [18:04:22,776][INFO][main][IgniteKernal] IGNITE_HOME=c:\Work\apache-ignite-2.7.0-SNAPSHOT-bin
> [18:04:22,776][INFO][main][IgniteKernal] VM arguments: [-XX:+AggressiveOpts, -Xms1g, -Xmx1g, -XX:MaxMetaspaceSize=256m,
> -DIGNITE_QUIET=false, -DIGNITE_SUCCESS_FILE=c:\Work\apache-ignite-2.7.0-SNAPSHOT-bin\work\ignite_success_37805c22-8b1e-4
> 14e-86e2-e3af2e5e67df, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=49148, -Dcom.sun.management.j
> mxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -DIGNITE_HOME=c:\Work\apache-ignite-2.7.0-SNAPSHO
> T-bin, -DIGNITE_PROG_NAME=ignite.bat, -DID=1]
> [18:04:22,776][INFO][main][IgniteKernal] System cache's DataRegion size is configured to 50 MB. Use DataStorageConfigura
> tion.systemRegionInitialSize property to change the setting.
> [18:04:22,776][INFO][main][IgniteKernal] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']]
> [18:04:22,776][WARNING][main][IgniteKernal] Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possib
> le problems in mixed environments.
> [18:04:22,776][WARNING][pub-#19][GridDiagnostic] This operating system has been tested less rigorously: Windows 10 10.0
> amd64. Our team will appreciate the feedback if you experience any problems running ignite in this environment.
> [18:04:22,776][INFO][main][IgniteKernal] 3-rd party licenses can be found at: c:\Work\apache-ignite-2.7.0-SNAPSHOT-bin\l
> ibs\licenses
> [18:04:22,829][INFO][main][IgnitePluginProcessor] Configured plugins:
> [18:04:22,829][INFO][main][IgnitePluginProcessor] ^-- None
> [18:04:22,829][INFO][main][IgnitePluginProcessor]
> [18:04:22,845][INFO][main][FailureProcessor] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=fals
> e, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED]]]]
> [18:04:22,883][INFO][main][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=47100, loc
> Host=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false]
> [18:04:22,884][WARNING][main][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs whe
> n running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides
> .
> [18:04:22,901][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi imp
> lementation)
> [18:04:22,919][WARNING][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon ar
> rival).
> [18:04:22,926][INFO][main][IgniteKernal] Security status [authentication=off, tls/ssl=off]
> [18:04:23,003][INFO][main][CacheObjectBinaryProcessorImpl] Resolved directory for serialized binary metadata: c:\Work\ap
> ache-ignite-2.7.0-SNAPSHOT-bin\work\binary_meta\1
> [18:04:23,125][INFO][main][FilePageStoreManager] Resolved page store work directory: c:\Work\apache-ignite-2.7.0-SNAPSHO
> T-bin\work\db\1
> [18:04:23,127][INFO][main][FileWriteAheadLogManager] Resolved write ahead log work directory: c:\Work\apache-ignite-2.7.
> 0-SNAPSHOT-bin\work\db\wal\1
> [18:04:23,128][INFO][main][FileWriteAheadLogManager] Resolved write ahead log archive directory: c:\Work\apache-ignite-2
> .7.0-SNAPSHOT-bin\work\db\wal\archive\1
> [18:04:23,415][INFO][main][FileWriteAheadLogManager] Started write-ahead log manager [mode=LOG_ONLY]
> [18:04:23,415][WARNING][main][GridCacheDatabaseSharedManager] Page eviction mode set for [dataRegion1] data will have no
> effect because the oldest pages are evicted automatically if Ignite persistence is enabled.
> [18:04:23,415][WARNING][main][GridCacheDatabaseSharedManager] Page eviction mode set for [dataRegion2] data will have no
> effect because the oldest pages are evicted automatically if Ignite persistence is enabled.
> [18:04:23,415][WARNING][main][GridCacheDatabaseSharedManager] DataRegionConfiguration.maxWalArchiveSize instead DataRegi
> onConfiguration.walHistorySize would be used for removing old archive wal files
> [18:04:23,462][INFO][main][PartitionsEvictManager] Evict partition permits=2
> [18:04:23,631][INFO][main][ClientListenerProcessor] Client connector processor has started on TCP port 10800
> [18:04:23,663][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=/127.0.0.1,
> port=11211]
> [18:04:23,763][INFO][main][IgniteKernal] Non-loopback local IPs: 172.25.4.52, fe80:0:0:0:2d2b:5bfb:e052:9fb0%eth5
> [18:04:23,763][INFO][main][IgniteKernal] Enabled local MACs: 54A050520B61
> [18:04:23,763][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint status [startMarker=null, endMarker=null]
> [18:04:23,779][INFO][main][PageMemoryImpl] Started page memory [memoryAllocated=100,0 MiB, pages=24808, tableSize=1,9 Mi
> B, checkpointBuffer=100,0 MiB]
> [18:04:23,779][INFO][main][GridCacheDatabaseSharedManager] Checking memory state [lastValidPos=FileWALPointer [idx=0, fi
> leOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000
> 000]
> [18:04:23,794][INFO][main][GridCacheDatabaseSharedManager] Applying lost cache updates since last checkpoint record [las
> tMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000]
> [18:04:23,794][INFO][main][GridCacheDatabaseSharedManager] Finished applying WAL changes [updatesApplied=0, time=0ms]
> [18:04:23,816][INFO][main][TcpDiscoverySpi] Connection check threshold is calculated: 10000
> [18:04:23,832][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port [port=47500, localHost=127.0.0.1/127.0.0.1, l
> ocNodeId=d02069db-6d0b-4a40-b185-1d95fa330853]
> [18:04:32,790][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1,
> rmtPort=61618]
> [18:04:32,822][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/12
> 7.0.0.1, rmtPort=61618]
> [18:04:32,822][INFO][tcp-disco-sock-reader-#4][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0
> .1:61618, rmtPort=61618]
> [18:04:32,839][INFO][tcp-disco-sock-reader-#4][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.
> 0.1:61618, rmtPort=61618
> [18:04:43,227][INFO][disco-notifier-worker-#41][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeI
> d=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]],
> crdNode=TcpDiscoveryNode [id=d02069db-6d0b-4a40-b185-1d95fa330853, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127
> .0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1540479863832, loc=true, ver=2.7.0#19700101-sha1:00
> 000000, isClient=false]]
> [18:04:43,310][INFO][main][IgniteKernal] Performance suggestions for grid (fix if possible)
> [18:04:43,310][INFO][main][IgniteKernal] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
> [18:04:43,310][INFO][main][IgniteKernal] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options)
> [18:04:43,310][INFO][main][IgniteKernal] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '
> -XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
> [18:04:43,310][INFO][main][IgniteKernal] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC'
> to JVM options)
> [18:04:43,310][INFO][main][IgniteKernal] Refer to this page for more performance suggestions: https://apacheignite.readm
> e.io/docs/jvm-and-system-tuning
> [18:04:43,310][INFO][main][IgniteKernal]
> [18:04:43,310][INFO][main][IgniteKernal] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
> [18:04:43,310][INFO][main][IgniteKernal] Data Regions Configured:
> [18:04:43,310][INFO][main][IgniteKernal] ^-- default [initSize=50,0 MiB, maxSize=100,0 MiB, persistence=true]
> [18:04:43,310][INFO][main][IgniteKernal] ^-- dataRegion1 [initSize=100,0 MiB, maxSize=200,0 MiB, persistence=true]
> [18:04:43,310][INFO][main][IgniteKernal] ^-- dataRegion2 [initSize=50,0 MiB, maxSize=100,0 MiB, persistence=true]
> [18:04:43,310][INFO][main][IgniteKernal]
> >>> +------------------------------------------+
> >>> Ignite ver. 2.7.0-SNAPSHOT#19700101-sha1:DEV
> >>> +------------------------------------------+
> >>> OS name: Windows 10 10.0 amd64
> >>> CPU(s): 8
> >>> Heap: 1.0GB
> >>> VM name: 836@work-pc
> >>> Local node [ID=D02069DB-6D0B-4A40-B185-1D95FA330853, order=1, clientMode=false]
> >>> Local node addresses: [127.0.0.1]
> >>> Local ports: TCP:10800 TCP:11211 TCP:47100 TCP:47500
> [18:04:43,310][INFO][main][IgniteKernal] >>> Ignite cluster is not active (limited functionality available). Use control
> .(sh|bat) script or IgniteCluster interface to activate.
> [18:04:43,310][INFO][main][GridDiscoveryManager] Topology snapshot [ver=1, locNode=d02069db, servers=1, clients=0, state
> =INACTIVE, CPUs=8, offheap=0.39GB, heap=1.0GB]
> [18:04:43,341][INFO][rest-#45][GridClusterStateProcessor] Sending activate request with BaselineTopology BaselineTopolog
> y [id=0, branchingHash=49, branchingType='New BaselineTopology', baselineNodes=[1]]
> [18:04:43,357][INFO][disco-notifier-worker-#41][GridClusterStateProcessor] Received activate request with BaselineTopolo
> gy[id=0]
> [18:04:43,357][INFO][disco-notifier-worker-#41][GridClusterStateProcessor] Started state transition: true
> [18:04:43,357][INFO][exchange-worker-#43][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTo
> pVer=1], mvccCrd=MvccCoordinator [nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, topVer=AffinityTopo
> logyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=true, crd=true, evt=DISCOVERY_CUSTOM_EVT, evtNode=d02069db-6d0b-4a
> 40-b185-1d95fa330853, customEvt=ChangeGlobalStateMessage [id=a6641cba661-606014c4-c468-4812-86de-7e8ae4cc30b9, reqId=a2c
> c8ce1-aff1-4600-aa51-a600474de991, initiatingNodeId=d02069db-6d0b-4a40-b185-1d95fa330853, activate=true, baselineTopolog
> y=BaselineTopology [id=0, branchingHash=49, branchingType='New BaselineTopology', baselineNodes=[1]], forceChangeBaselin
> eTopology=false, timestamp=1540479883341], allowMerge=false]
> [18:04:43,357][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Start activation process [nodeId=d02069db-6d0
> b-4a40-b185-1d95fa330853, client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
> [18:04:43,357][INFO][exchange-worker-#43][FilePageStoreManager] Resolved page store work directory: c:\Work\apache-ignit
> e-2.7.0-SNAPSHOT-bin\work\db\1
> [18:04:43,357][INFO][exchange-worker-#43][FileWriteAheadLogManager] Resolved write ahead log work directory: c:\Work\apa
> che-ignite-2.7.0-SNAPSHOT-bin\work\db\wal\1
> [18:04:43,372][INFO][exchange-worker-#43][FileWriteAheadLogManager] Resolved write ahead log archive directory: c:\Work\
> apache-ignite-2.7.0-SNAPSHOT-bin\work\db\wal\archive\1
> [18:04:43,372][INFO][exchange-worker-#43][FileWriteAheadLogManager] Started write-ahead log manager [mode=LOG_ONLY]
> [18:04:43,379][INFO][exchange-worker-#43][PageMemoryImpl] Started page memory [memoryAllocated=200,0 MiB, pages=49624, t
> ableSize=3,9 MiB, checkpointBuffer=200,0 MiB]
> [18:04:43,379][INFO][exchange-worker-#43][PageMemoryImpl] Started page memory [memoryAllocated=100,0 MiB, pages=24808, t
> ableSize=1,9 MiB, checkpointBuffer=100,0 MiB]
> [18:04:43,379][INFO][exchange-worker-#43][PageMemoryImpl] Started page memory [memoryAllocated=100,0 MiB, pages=24808, t
> ableSize=1,9 MiB, checkpointBuffer=100,0 MiB]
> [18:04:43,379][INFO][exchange-worker-#43][PageMemoryImpl] Started page memory [memoryAllocated=100,0 MiB, pages=24808, t
> ableSize=1,9 MiB, checkpointBuffer=100,0 MiB]
> [18:04:43,379][INFO][exchange-worker-#43][PageMemoryImpl] Started page memory [memoryAllocated=100,0 MiB, pages=24808, t
> ableSize=1,9 MiB, checkpointBuffer=100,0 MiB]
> [18:04:43,379][INFO][exchange-worker-#43][PageMemoryImpl] Started page memory [memoryAllocated=100,0 MiB, pages=24808, t
> ableSize=1,9 MiB, checkpointBuffer=100,0 MiB]
> [18:04:43,395][INFO][exchange-worker-#43][msg] Components activation performed in 38 ms.
> [18:04:43,410][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Read checkpoint status [startMarker=null, endM
> arker=null]
> [18:04:43,410][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Checking memory state [lastValidPos=FileWALPoi
> nter [idx=0, fileOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000
> -0000-000000000000]
> [18:04:43,410][INFO][exchange-worker-#43][FileWriteAheadLogManager] Resuming logging to WAL segment [file=c:\Work\apache
> -ignite-2.7.0-SNAPSHOT-bin\work\db\wal\1\0000000000000000.wal, offset=0, ver=2]
> [18:04:43,457][INFO][exchange-worker-#43][GridClusterStateProcessor] Writing BaselineTopology[id=0]
> [18:04:43,473][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Binary recovery performed in 78 ms.
> [18:04:43,473][INFO][exchange-worker-#43][CacheAffinitySharedManager] Updating caches registry performed in 0 ms.
> [18:04:43,611][INFO][exchange-worker-#43][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, memo
> ryPolicyName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false], encryptionEnabled=fal
> se]
> [18:04:43,611][INFO][exchange-worker-#43][CacheAffinitySharedManager] Caches starting performed in 138 ms.
> [18:04:43,611][INFO][exchange-worker-#43][CacheAffinitySharedManager] Affinity initialization for started caches perform
> ed in 0 ms.
> [18:04:43,611][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Successfully activated caches [nodeId=d02069d
> b-6d0b-4a40-b185-1d95fa330853, client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
> [18:04:43,626][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future
> [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=LOCAL]
> [18:04:43,626][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Read checkpoint status [startMarker=null, endM
> arker=null]
> [18:04:43,626][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Applying lost cache updates since last checkpo
> int record [lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000]
> [18:04:43,679][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Restoring partition state for local groups [cn
> tPartStateWal=0, lastCheckpointId=00000000-0000-0000-0000-000000000000]
> [18:04:43,695][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Finished restoring partition state for local g
> roups [cntProcessed=100, cntPartStateWal=0, time=16ms]
> [18:04:43,695][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Finished applying WAL changes [updatesApplied=
> 0, time=69ms]
> [18:04:43,695][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Logical recovery performed in 69 ms.
> [18:04:43,726][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Checkpointer initilialzation performed in 15 m
> s.
> [18:04:43,726][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityT
> opologyVersion [topVer=1, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
> [18:04:43,726][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 0 m
> s.
> [18:04:43,726][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms.
> [18:04:43,726][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Partitions assignment performed in 0 ms.
> [18:04:43,726][INFO][exchange-worker-#43][CacheAffinitySharedManager] Affinity recalculation (custom message) performed
> in 0 ms.
> [18:04:43,742][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 16 ms.
> [18:04:43,742][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed i
> n 0 ms.
> [18:04:43,742][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopo
> logyVersion [topVer=1, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], err=null]
> [18:04:43,742][INFO][disco-notifier-worker-#41][GridClusterStateProcessor] Received state change finish message: true
> [18:04:43,742][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] Sending Full Message for AffinityTopologyVe
> rsion [topVer=1, minorTopVer=1] performed in 0 ms.
> [18:04:43,742][INFO][exchange-worker-#43][MvccProcessorImpl] Initialize local node as mvcc coordinator [node=d02069db-6d
> 0b-4a40-b185-1d95fa330853, crdVer=1540479883228]
> [18:04:43,742][INFO][exchange-worker-#43][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, l
> ocalNodeId=d02069db-6d0b-4a40-b185-1d95fa330853
> [18:04:43,758][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=d0206
> 9db-6d0b-4a40-b185-1d95fa330853, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, min
> orTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=d02069db-6d0b-4a40-b185-1d95fa330853, addrs=ArrayLis
> t [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1540479863832
> , loc=true, ver=2.7.0#19700101-sha1:00000000, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=1, min
> orTopVer=1], durationFromInit=401]
> [18:04:43,758][INFO][exchange-worker-#43][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=1, minorT
> opVer=1], crd=true]
> [18:04:43,758][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [t
> op=AffinityTopologyVersion [topVer=1, minorTopVer=1], force=false, evt=DISCOVERY_CUSTOM_EVT, node=d02069db-6d0b-4a40-b18
> 5-1d95fa330853]
> [18:04:43,758][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=143137c0
> -86bf-44a0-9f9c-efba0161e74d, startPtr=FileWALPointer [idx=0, fileOff=119489, len=1947], checkpointLockWait=0ms, checkpo
> intLockHoldTime=15ms, walCpRecordFsyncDuration=16ms, pages=17, reason='timeout']
> [18:04:43,773][INFO][sys-#54][GridClusterStateProcessor] Successfully performed final activation steps [nodeId=d02069db-
> 6d0b-4a40-b185-1d95fa330853, client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
> [18:04:43,795][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=143137c0-86bf-4
> 4a0-9f9c-efba0161e74d, pages=17, markPos=FileWALPointer [idx=0, fileOff=119489, len=1947], walSegmentsCleared=0, walSegm
> entsCovered=[], markDuration=62ms, pagesWrite=0ms, fsync=22ms, total=84ms]
> [18:04:44,729][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modifi
> ed) [checkpointLockWait=0ms, checkpointLockHoldTime=0ms, reason='timeout']
> [18:04:48,037][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1,
> rmtPort=61665]
> [18:04:48,037][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/12
> 7.0.0.1, rmtPort=61665]
> [18:04:48,041][INFO][tcp-disco-sock-reader-#5][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0
> .1:61665, rmtPort=61665]
> [18:04:48,063][INFO][tcp-disco-sock-reader-#5][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.
> 0.1:61665, rmtPort=61665
> [18:04:48,069][INFO][tcp-disco-msg-worker-#2][GridEncryptionManager] Joining node doesn't have encryption data [node=5df
> 37010-aaf1-4c1c-88fb-53b8b178c4f5]
> [18:04:48,093][INFO][tcp-disco-msg-worker-#2][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=5df37010-aaf1
> -4c1c-88fb-53b8b178c4f5, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=0, int
> Order=2, lastExchangeTime=1540479888049, loc=false, ver=2.7.0#19700101-sha1:00000000, isClient=false]]
> [18:04:48,129][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1,
> rmtPort=61667]
> [18:04:48,129][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/12
> 7.0.0.1, rmtPort=61667]
> [18:04:48,130][INFO][tcp-disco-sock-reader-#6][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0
> .1:61667, rmtPort=61667]
> [18:04:48,136][INFO][disco-event-worker-#42][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=5df3
> 7010-aaf1-4c1c-88fb-53b8b178c4f5, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, ord
> er=2, intOrder=2, lastExchangeTime=1540479888049, loc=false, ver=2.7.0#19700101-sha1:00000000, isClient=false]
> [18:04:48,137][INFO][disco-event-worker-#42][GridDiscoveryManager] Topology snapshot [ver=2, locNode=d02069db, servers=2
> , clients=0, state=ACTIVE, CPUs=8, offheap=0.78GB, heap=2.0GB]
> [18:04:48,137][INFO][disco-event-worker-#42][GridDiscoveryManager] ^-- Baseline [id=0, size=1, online=1, offline=0]
> [18:04:48,138][INFO][exchange-worker-#43][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=2, minorTo
> pVer=0], mvccCrd=MvccCoordinator [nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, topVer=AffinityTopo
> logyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_JOINED, evtNode=5df37010-aaf1-4c1c-88fb-
> 53b8b178c4f5, customEvt=null, allowMerge=true]
> [18:04:48,145][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future
> [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED]
> [18:04:48,148][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch
> : ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [t
> opVer=2, minorTopVer=0]]]
> [18:04:48,148][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future
> [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL]
> [18:04:48,148][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Logical recovery performed in 0 ms.
> [18:04:48,149][INFO][exchange-worker-#43][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=2, minorT
> opVer=0], crd=true]
> [18:04:48,725][INFO][grid-nio-worker-tcp-comm-0-#24][TcpCommunicationSpi] Accepted incoming communication connection [lo
> cAddr=/127.0.0.1:47100, rmtAddr=/127.0.0.1:61669]
> [18:04:48,741][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopology
> Version [topVer=2, minorTopVer=0], node=5df37010-aaf1-4c1c-88fb-53b8b178c4f5, allReceived=true]
> [18:04:48,741][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=Affinit
> yTopologyVersion [topVer=2, minorTopVer=0]]
> [18:04:48,741][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 0 ms.
> [18:04:48,741][INFO][sys-#56][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersi
> on [topVer=2, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]]
> [18:04:48,741][INFO][sys-#56][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 0 ms.
> [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 15 ms.
> [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms.
> [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Partitions assignment performed in 0 ms.
> [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 0 ms.
> [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 0 ms.
> [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 0 ms.
> [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion
> [topVer=2, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], err=null]
> [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms.
> [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=d02069db-6d0b-4a4
> 0-b185-1d95fa330853, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0],
> evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=5df37010-aaf1-4c1c-88fb-53b8b178c4f5, addrs=ArrayList [127.0.0.1], sockAd
> drs=HashSet [/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540479888049, loc=false, ver=2.7.
> 0#19700101-sha1:00000000, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], duratio
> nFromInit=620]
> [18:04:48,756][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [t
> op=AffinityTopologyVersion [topVer=2, minorTopVer=0], force=false, evt=NODE_JOINED, node=5df37010-aaf1-4c1c-88fb-53b8b17
> 8c4f5]
> [18:04:49,769][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modifi
> ed) [checkpointLockWait=0ms, checkpointLockHoldTime=3ms, reason='timeout']
> [18:04:55,817][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modifi
> ed) [checkpointLockWait=0ms, checkpointLockHoldTime=1ms, reason='timeout']
> [18:05:02,884][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modifi
> ed) [checkpointLockWait=0ms, checkpointLockHoldTime=2ms, reason='timeout']
> [18:05:09,804][INFO][exchange-worker-#43][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=2, minorTo
> pVer=1], mvccCrd=MvccCoordinator [nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, topVer=AffinityTopo
> logyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=DISCOVERY_CUSTOM_EVT, evtNode=d02069db-6d0b-4
> a40-b185-1d95fa330853, customEvt=DynamicCacheChangeBatch [id=c6741cba661-606014c4-c468-4812-86de-7e8ae4cc30b9, reqs=Arra
> yList [DynamicCacheChangeRequest [cacheName=SQL_PUBLIC_T1, hasCfg=true, nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, cli
> entStartOnly=false, stop=false, destroy=false, disabledAfterStartfalse]], exchangeActions=ExchangeActions [startCaches=[
> SQL_PUBLIC_T1], stopCaches=null, startGrps=[SQL_PUBLIC_T1], stopGrps=[], resetParts=null, stateChangeRequest=null], star
> tCaches=false], allowMerge=false]
> [18:05:09,804][INFO][exchange-worker-#43][CacheAffinitySharedManager] Updating caches registry performed in 0 ms.
> [18:05:09,875][INFO][exchange-worker-#43][MvccProcessorImpl] Mvcc processor started.
> [18:05:09,912][INFO][exchange-worker-#43][GridCacheProcessor] Started cache [name=SQL_PUBLIC_T1, id=-1691533662, memoryP
> olicyName=default, mode=PARTITIONED, atomicity=TRANSACTIONAL_SNAPSHOT, backups=1, mvcc=true], encryptionEnabled=false]
> [18:05:09,912][INFO][exchange-worker-#43][CacheAffinitySharedManager] Caches starting performed in 108 ms.
> [18:05:09,915][INFO][exchange-worker-#43][GridAffinityAssignmentCache] Local node affinity assignment distribution is no
> t ideal [cache=SQL_PUBLIC_T1, expectedPrimary=1024,00, actualPrimary=1024, expectedBackups=1024,00, actualBackups=0, war
> ningThreshold=50,00%]
> [18:05:09,916][INFO][exchange-worker-#43][CacheAffinitySharedManager] Affinity initialization for started caches perform
> ed in 3 ms.
> [18:05:09,917][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future
> [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED]
> [18:05:09,934][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=39c009dc
> -7bd4-465e-8a4d-156ea670668b, startPtr=FileWALPointer [idx=0, fileOff=131199, len=1953], checkpointLockWait=5ms, checkpo
> intLockHoldTime=1ms, walCpRecordFsyncDuration=9ms, pages=13, reason='timeout']
> [18:05:09,945][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch
> : ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [t
> opVer=2, minorTopVer=1]]]
> [18:05:09,946][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future
> [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=LOCAL]
> [18:05:09,947][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=39c009dc-7bd4-4
> 65e-8a4d-156ea670668b, pages=13, markPos=FileWALPointer [idx=0, fileOff=131199, len=1953], walSegmentsCleared=0, walSegm
> entsCovered=[], markDuration=22ms, pagesWrite=2ms, fsync=11ms, total=40ms]
> [18:05:09,963][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Logical recovery performed in 17 ms.
> [18:05:10,014][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=Affi
> nityTopologyVersion [topVer=2, minorTopVer=1], node=5df37010-aaf1-4c1c-88fb-53b8b178c4f5, allReceived=true]
> [18:05:10,014][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityT
> opologyVersion [topVer=2, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]]
> [18:05:10,015][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 0 m
> s.
> [18:05:10,016][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Partitions validation performed in 1 ms.
> [18:05:10,020][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Partitions assignment performed in 4 ms.
> [18:05:10,025][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 4 ms.
> [18:05:10,025][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 0 ms.
> [18:05:10,025][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed i
> n 0 ms.
> [18:05:10,026][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopo
> logyVersion [topVer=2, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], err=null]
> [18:05:10,029][INFO][exchange-worker-#43][GridCacheProcessor] Finish proxy initialization, cacheName=SQL_PUBLIC_T1, loca
> lNodeId=d02069db-6d0b-4a40-b185-1d95fa330853
> [18:05:10,030][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=d0206
> 9db-6d0b-4a40-b185-1d95fa330853, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=2, min
> orTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=d02069db-6d0b-4a40-b185-1d95fa330853, addrs=ArrayLis
> t [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1540479909804
> , loc=true, ver=2.7.0#19700101-sha1:00000000, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=2, min
> orTopVer=1], durationFromInit=222]
> [18:05:10,030][INFO][exchange-worker-#43][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=2, minorT
> opVer=1], crd=true]
> [18:05:10,031][INFO][pub-#70][GridCacheDatabaseSharedManager] Finished indexes rebuilding for cache [name=SQL_PUBLIC_T1,
> grpName=null]
> [18:05:10,033][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [t
> op=AffinityTopologyVersion [topVer=2, minorTopVer=1], force=false, evt=DISCOVERY_CUSTOM_EVT, node=d02069db-6d0b-4a40-b18
> 5-1d95fa330853]
> [18:05:10,919][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modifi
> ed) [checkpointLockWait=0ms, checkpointLockHoldTime=4ms, reason='timeout']
> [18:05:14,456][INFO][disco-event-worker-#42][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=5df37010-aaf
> 1-4c1c-88fb-53b8b178c4f5, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=2, in
> tOrder=2, lastExchangeTime=1540479888049, loc=false, ver=2.7.0#19700101-sha1:00000000, isClient=false]
> [18:05:14,456][INFO][disco-event-worker-#42][GridDiscoveryManager] Topology snapshot [ver=3, locNode=d02069db, servers=1
> , clients=0, state=ACTIVE, CPUs=8, offheap=0.39GB, heap=1.0GB]
> [18:05:14,456][INFO][disco-event-worker-#42][GridDiscoveryManager] ^-- Baseline [id=0, size=1, online=1, offline=0]
> [18:05:14,488][INFO][exchange-worker-#43][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=3, minorTo
> pVer=0], mvccCrd=MvccCoordinator [nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, topVer=AffinityTopo
> logyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_LEFT, evtNode=5df37010-aaf1-4c1c-88fb-53
> b8b178c4f5, customEvt=null, allowMerge=true]
> [18:05:14,488][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future
> [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED]
> [18:05:14,488][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch
> : ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [t
> opVer=3, minorTopVer=0]]]
> [18:05:14,488][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future
> [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL]
> [18:05:14,488][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Logical recovery performed in 0 ms.
> [18:05:14,488][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge
> [ver=AffinityTopologyVersion [topVer=3, minorTopVer=0]]
> [18:05:14,488][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 0 ms.
> [18:05:14,488][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityT
> opologyVersion [topVer=3, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]]
> [18:05:14,488][INFO][exchange-worker-#43][GridAffinityAssignmentCache] Local node affinity assignment distribution is no
> t ideal [cache=SQL_PUBLIC_T1, expectedPrimary=1024,00, actualPrimary=1024, expectedBackups=1024,00, actualBackups=0, war
> ningThreshold=50,00%]
> [18:05:14,511][INFO][exchange-worker-#43][CacheAffinitySharedManager] Affinity recalculation (on server left) performed
> in 23 ms.
> [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 25
> ms.
> [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms.
> [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms.
> [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 0 ms.
> [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed i
> n 0 ms.
> [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopo
> logyVersion [topVer=3, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], err=null]
> [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms.
> [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=d0206
> 9db-6d0b-4a40-b185-1d95fa330853, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=3, min
> orTopVer=0], evt=NODE_LEFT, evtNode=TcpDiscoveryNode [id=5df37010-aaf1-4c1c-88fb-53b8b178c4f5, addrs=ArrayList [127.0.0.
> 1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1540479888049, loc=false
> , ver=2.7.0#19700101-sha1:00000000, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0
> ], durationFromInit=15]
> [18:05:14,513][INFO][exchange-worker-#43][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=3, minorT
> opVer=0], crd=true]
> [18:05:14,513][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [t
> op=AffinityTopologyVersion [topVer=3, minorTopVer=0], force=false, evt=NODE_LEFT, node=5df37010-aaf1-4c1c-88fb-53b8b178c
> 4f5]
> [18:05:14,982][INFO][tcp-disco-sock-reader-#6][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.
> 0.1:61667, rmtPort=61667
> [18:05:21,967][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modifi
> ed) [checkpointLockWait=0ms, checkpointLockHoldTime=5ms, reason='timeout']
> [18:05:23,818][INFO][exchange-worker-#43][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=3, minorTo
> pVer=1], mvccCrd=MvccCoordinator [nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, topVer=AffinityTopo
> logyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=DISCOVERY_CUSTOM_EVT, evtNode=d02069db-6d0b-4
> a40-b185-1d95fa330853, customEvt=DynamicCacheChangeBatch [id=ea741cba661-606014c4-c468-4812-86de-7e8ae4cc30b9, reqs=Arra
> yList [DynamicCacheChangeRequest [cacheName=SQL_PUBLIC_T1, hasCfg=false, nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, cl
> ientStartOnly=false, stop=true, destroy=false, disabledAfterStartfalse]], exchangeActions=ExchangeActions [startCaches=n
> ull, stopCaches=[SQL_PUBLIC_T1], startGrps=[], stopGrps=[SQL_PUBLIC_T1, destroy=true], resetParts=null, stateChangeReque
> st=null], startCaches=false], allowMerge=false]
> [18:05:23,819][INFO][exchange-worker-#43][CacheAffinitySharedManager] Updating caches registry performed in 1 ms.
> [18:05:23,820][INFO][exchange-worker-#43][CacheAffinitySharedManager] Caches starting performed in 0 ms.
> [18:05:23,820][INFO][exchange-worker-#43][CacheAffinitySharedManager] Affinity initialization for started caches perform
> ed in 0 ms.
> [18:05:23,820][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future
> [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED]
> [18:05:23,821][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch
> : ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [t
> opVer=3, minorTopVer=1]]]
> [18:05:23,821][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future
> [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=LOCAL]
> [18:05:23,821][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Logical recovery performed in 0 ms.
> [18:05:23,821][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityT
> opologyVersion [topVer=3, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=3, minorTopVer=1]]
> [18:05:23,821][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 0 m
> s.
> [18:05:23,821][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms.
> [18:05:23,822][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Partitions assignment performed in 0 ms.
> [18:05:23,825][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 3 ms.
> [18:05:23,825][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed i
> n 0 ms.
> [18:05:23,825][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopo
> logyVersion [topVer=3, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], err=null]
> [18:05:23,827][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] Skipping checkpoint (no pages were modifi
> ed) [checkpointLockWait=0ms, checkpointLockHoldTime=2ms, reason='caches stop']
> [18:05:23,833][INFO][exchange-worker-#43][GridCacheProcessor] Stopped cache [cacheName=SQL_PUBLIC_T1]
> [18:05:23,851][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=d0206
> 9db-6d0b-4a40-b185-1d95fa330853, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=3, min
> orTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=d02069db-6d0b-4a40-b185-1d95fa330853, addrs=ArrayLis
> t [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1540479914456
> , loc=true, ver=2.7.0#19700101-sha1:00000000, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=3, min
> orTopVer=1], durationFromInit=33]
> [18:05:23,851][INFO][exchange-worker-#43][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=3, minorT
> opVer=1], crd=true]
> [18:05:23,853][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [t
> op=AffinityTopologyVersion [topVer=3, minorTopVer=1], force=false, evt=DISCOVERY_CUSTOM_EVT, node=d02069db-6d0b-4a40-b18
> 5-1d95fa330853]
> [18:05:36,638][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1,
> rmtPort=61721]
> [18:05:36,638][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/12
> 7.0.0.1, rmtPort=61721]
> [18:05:36,638][INFO][tcp-disco-sock-reader-#7][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0
> .1:61721, rmtPort=61721]
> [18:05:36,669][INFO][tcp-disco-sock-reader-#7][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.
> 0.1:61721, rmtPort=61721
> [18:05:36,685][INFO][tcp-disco-msg-worker-#2][GridEncryptionManager] Joining node doesn't have encryption data [node=d07
> 9c11d-bd7d-4d65-93ab-dee8d0b280e3]
> [18:05:36,685][INFO][tcp-disco-msg-worker-#2][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=d079c11d-bd7d
> -4d65-93ab-dee8d0b280e3, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=0, int
> Order=3, lastExchangeTime=1540479936669, loc=false, ver=2.7.0#19700101-sha1:00000000, isClient=false]]
> [18:05:36,738][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1,
> rmtPort=61723]
> [18:05:36,738][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/12
> 7.0.0.1, rmtPort=61723]
> [18:05:36,738][INFO][tcp-disco-sock-reader-#8][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0
> .1:61723, rmtPort=61723]
> [18:05:36,754][INFO][disco-event-worker-#42][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=d079
> c11d-bd7d-4d65-93ab-dee8d0b280e3, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, ord
> er=4, intOrder=3, lastExchangeTime=1540479936669, loc=false, ver=2.7.0#19700101-sha1:00000000, isClient=false]
> [18:05:36,754][INFO][disco-event-worker-#42][GridDiscoveryManager] Topology snapshot [ver=4, locNode=d02069db, servers=2
> , clients=0, state=ACTIVE, CPUs=8, offheap=0.78GB, heap=2.0GB]
> [18:05:36,754][INFO][disco-event-worker-#42][GridDiscoveryManager] ^-- Baseline [id=0, size=1, online=1, offline=0]
> [18:05:36,754][INFO][exchange-worker-#43][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTo
> pVer=0], mvccCrd=MvccCoordinator [nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, topVer=AffinityTopo
> logyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_JOINED, evtNode=d079c11d-bd7d-4d65-93ab-
> dee8d0b280e3, customEvt=null, allowMerge=true]
> [18:05:36,754][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future
> [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED]
> [18:05:36,754][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch
> : ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [t
> opVer=4, minorTopVer=0]]]
> [18:05:36,754][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future
> [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL]
> [18:05:36,754][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] Logical recovery performed in 0 ms.
> [18:05:36,754][INFO][exchange-worker-#43][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=4, minorT
> opVer=0], crd=true]
> [18:05:37,423][INFO][grid-nio-worker-tcp-comm-1-#25][TcpCommunicationSpi] Accepted incoming communication connection [lo
> cAddr=/127.0.0.1:47100, rmtAddr=/127.0.0.1:61724]
> [18:05:37,439][INFO][sys-#62][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopology
> Version [topVer=4, minorTopVer=0], node=d079c11d-bd7d-4d65-93ab-dee8d0b280e3, allReceived=true]
> [18:05:37,439][SEVERE][sys-#62][GridDhtPartitionsExchangeFuture] Failed to notify listener: o.a.i.i.processors.cache.dis
> tributed.dht.preloader.GridDhtPartitionsExchangeFuture$2@3e21b701
> java.lang.NullPointerException
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.clientTopology(GridCachePartiti
> onExchangeManager.java:835)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updateP
> artitionSingleMap(GridDhtPartitionsExchangeFuture.java:4031)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.process
> SingleMessage(GridDhtPartitionsExchangeFuture.java:2790)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$
> 100(GridDhtPartitionsExchangeFuture.java:139)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply
> (GridDhtPartitionsExchangeFuture.java:2612)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply
> (GridDhtPartitionsExchangeFuture.java:2600)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:355)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onRecei
> veSingleMessage(GridDhtPartitionsExchangeFuture.java:2600)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(Gr
> idCachePartitionExchangeManager.java:1656)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionE
> xchangeManager.java:142)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionE
> xchangeManager.java:375)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionE
> xchangeManager.java:333)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCacheP
> artitionExchangeManager.java:3102)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCacheP
> artitionExchangeManager.java:3081)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:119
> 7)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [18:05:37,455][SEVERE][sys-#62][GridCacheIoManager] Failed processing message [senderId=d079c11d-bd7d-4d65-93ab-dee8d0b2
> 80e3, msg=GridDhtPartitionsSingleMessage [parts=HashMap {-2100569601=GridDhtPartitionMap [moving=0, top=AffinityTopology
> Version [topVer=-1, minorTopVer=0], updateSeq=2, size=0], -1691533662=GridDhtPartitionMap [moving=0, top=AffinityTopolog
> yVersion [topVer=4, minorTopVer=0], updateSeq=2, size=0]}, partCntrs=HashMap {-2100569601=CachePartitionPartialCountersM
> ap {}, -1691533662=CachePartitionPartialCountersMap {}}, partsSizes=null, partHistCntrs=null, err=null, client=false, co
> mpress=false, finishMsg=null, activeQryTrackers=GridLongList [idx=0, arr=[]], super=GridDhtPartitionsAbstractMessage [ex
> chId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], discoEvt=null, nodeId=d079c11
> d, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0, order=1540479929882, nodeOrder=0], super=GridCacheMessage [msgI
> d=1, depInfo=null, err=null, skipPrepare=false]]]]
> java.lang.NullPointerException
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.clientTopology(GridCachePartiti
> onExchangeManager.java:835)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updateP
> artitionSingleMap(GridDhtPartitionsExchangeFuture.java:4031)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.process
> SingleMessage(GridDhtPartitionsExchangeFuture.java:2790)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$
> 100(GridDhtPartitionsExchangeFuture.java:139)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply
> (GridDhtPartitionsExchangeFuture.java:2612)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply
> (GridDhtPartitionsExchangeFuture.java:2600)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:355)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onRecei
> veSingleMessage(GridDhtPartitionsExchangeFuture.java:2600)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(Gr
> idCachePartitionExchangeManager.java:1656)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionE
> xchangeManager.java:142)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionE
> xchangeManager.java:375)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionE
> xchangeManager.java:333)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCacheP
> artitionExchangeManager.java:3102)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCacheP
> artitionExchangeManager.java:3081)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:119
> 7)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [18:05:37,455][SEVERE][sys-#62][] Critical system error detected. Will be handled accordingly to configured handler [hnd
> =StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableS
> et [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.NullPointerException]]
> java.lang.NullPointerException
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.clientTopology(GridCachePartiti
> onExchangeManager.java:835)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updateP
> artitionSingleMap(GridDhtPartitionsExchangeFuture.java:4031)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.process
> SingleMessage(GridDhtPartitionsExchangeFuture.java:2790)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$
> 100(GridDhtPartitionsExchangeFuture.java:139)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply
> (GridDhtPartitionsExchangeFuture.java:2612)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply
> (GridDhtPartitionsExchangeFuture.java:2600)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:355)
> at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onRecei
> veSingleMessage(GridDhtPartitionsExchangeFuture.java:2600)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(Gr
> idCachePartitionExchangeManager.java:1656)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionE
> xchangeManager.java:142)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionE
> xchangeManager.java:375)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionE
> xchangeManager.java:333)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCacheP
> artitionExchangeManager.java:3102)
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCacheP
> artitionExchangeManager.java:3081)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:119
> 7)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> [18:05:37,455][WARNING][sys-#62][FailureProcessor] No deadlocked threads detected.
> [18:05:37,486][WARNING][sys-#62][FailureProcessor] Thread dump at 2018/10/25 18:05:37 MSK
> Thread [name="tcp-disco-sock-reader-#8", id=112, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked java.io.BufferedInputStream@69777ba6
> at o.a.i.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
> at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2663)
> at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2679)
> at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
> at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
> at o.a.i.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:43)
> at o.a.i.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:137)
> at o.a.i.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
> at o.a.i.i.util.IgniteUtils.unmarshal(IgniteUtils.java:10085)
> at o.a.i.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:6246)
> at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Thread [name="pool-3-thread-1", id=110, state=TIMED_WAITING, blockCnt=0, waitCnt=2]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@36a3688e, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="client-connector-#73", id=109, state=WAITING, blockCnt=0, waitCnt=2]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54dbe05e, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="pub-#72", id=108, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33f5d231, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="pub-#70", id=106, state=TIMED_WAITING, blockCnt=0, waitCnt=2]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33f5d231, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#69", id=105, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#68", id=104, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="vacuum-cleaner-#66", id=102, state=WAITING, blockCnt=0, waitCnt=3]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4cc7d605, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at o.a.i.i.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.body(MvccProcessorImpl.java:2184)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="vacuum-cleaner-#65", id=101, state=WAITING, blockCnt=2, waitCnt=10]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4cc7d605, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at o.a.i.i.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.body(MvccProcessorImpl.java:2184)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="vacuum-scheduler-#64", id=100, state=TIMED_WAITING, blockCnt=0, waitCnt=8]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7774)
> at o.a.i.i.processors.cache.mvcc.MvccProcessorImpl$VacuumScheduler.body(MvccProcessorImpl.java:2158)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="query-#63", id=99, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@479516d3, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#62", id=98, state=RUNNABLE, blockCnt=0, waitCnt=1]
> at sun.management.ThreadImpl.dumpThreads0(Native Method)
> at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
> at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1364)
> at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:132)
> - locked o.a.i.i.processors.failure.FailureProcessor@5a6c304a
> at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:108)
> at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1071)
> at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> at o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked synchronizers:
> java.util.concurrent.ThreadPoolExecutor$Worker@69455d2e
> Thread [name="client-connector-#61", id=97, state=WAITING, blockCnt=0, waitCnt=2]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54dbe05e, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#60", id=96, state=TIMED_WAITING, blockCnt=0, waitCnt=2]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#59", id=95, state=TIMED_WAITING, blockCnt=0, waitCnt=2]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="jdbc-request-handler-worker-#58", id=94, state=WAITING, blockCnt=0, waitCnt=3]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@52c43e80, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at o.a.i.i.processors.odbc.jdbc.JdbcRequestHandlerWorker.body(JdbcRequestHandlerWorker.java:85)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="client-connector-#57", id=93, state=WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54dbe05e, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#56", id=90, state=TIMED_WAITING, blockCnt=0, waitCnt=2]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="srvc-deploy-#55", id=87, state=WAITING, blockCnt=0, waitCnt=6]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20c1ad23, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#54", id=86, state=TIMED_WAITING, blockCnt=3, waitCnt=2]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="checkpoint-runner-#53", id=85, state=WAITING, blockCnt=4, waitCnt=781]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f8d1466, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="checkpoint-runner-#52", id=84, state=WAITING, blockCnt=6, waitCnt=797]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f8d1466, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="checkpoint-runner-#51", id=83, state=WAITING, blockCnt=3, waitCnt=823]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f8d1466, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="checkpoint-runner-#50", id=82, state=WAITING, blockCnt=4, waitCnt=804]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f8d1466, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="db-checkpoint-thread-#49", id=81, state=TIMED_WAITING, blockCnt=4, waitCnt=225]
> Lock [object=o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer@745883bd, ownerName=nu
> ll, ownerId=-1]
> at java.lang.Object.wait(Native Method)
> at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.waitCheckpointEvent(GridCach
> eDatabaseSharedManager.java:3537)
> at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseShared
> Manager.java:3085)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-#48", id=80, state=TIMED_WAITING, blockCnt=0, waitCnt=2]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="wal-segment-syncer-#47", id=79, state=TIMED_WAITING, blockCnt=0, waitCnt=108]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7774)
> at o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$WalSegmentSyncer.body(FileWriteAheadLogMana
> ger.java:3514)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="wal-file-archiver%null-#46", id=78, state=WAITING, blockCnt=0, waitCnt=1]
> Lock [object=o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage@79b9388, ownerName=null, owne
> rId=-1]
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage.awaitSegment(SegmentCurrentStateSto
> rage.java:73)
> at o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage.waitNextSegmentForArchivation(Segme
> ntCurrentStateStorage.java:90)
> at o.a.i.i.processors.cache.persistence.wal.aware.SegmentAware.waitNextSegmentForArchivation(SegmentAware.java:7
> 9)
> at o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.body(FileWriteAheadLogManager.
> java:1719)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="rest-#45", id=77, state=TIMED_WAITING, blockCnt=1, waitCnt=2]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a9e8eed, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="rest-#44", id=76, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a9e8eed, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="session-timeout-worker-#34", id=51, state=TIMED_WAITING, blockCnt=0, waitCnt=54]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.processors.rest.GridRestProcessor$4.body(GridRestProcessor.java:495)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="exchange-worker-#43", id=74, state=TIMED_WAITING, blockCnt=2, waitCnt=35]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
> at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:219)
> at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160)
> at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeMan
> ager.java:2695)
> at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeMana
> ger.java:2539)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="ignite-update-notifier-timer", id=73, state=TIMED_WAITING, blockCnt=1, waitCnt=3]
> Lock [object=java.util.TaskQueue@d1791fc, ownerName=null, ownerId=-1]
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:552)
> at java.util.TimerThread.run(Timer.java:505)
> Thread [name="upd-ver-checker", id=72, state=TIMED_WAITING, blockCnt=0, waitCnt=11]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.processors.cluster.GridUpdateNotifier$1.run(GridUpdateNotifier.java:115)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="disco-event-worker-#42", id=71, state=WAITING, blockCnt=0, waitCnt=38]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d62be88, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2856)
> at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2825)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="tcp-disco-srvr-#3", id=69, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at java.net.DualStackPlainSocketImpl.accept0(Native Method)
> at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
> - locked java.net.SocksSocketImpl@66be6d7f
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5845)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at o.a.i.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5763)
> at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Thread [name="tcp-disco-msg-worker-#2", id=68, state=TIMED_WAITING, blockCnt=6, waitCnt=5916]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@38097ea3, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
> at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
> at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7183)
> at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Thread [name="disco-notifier-worker-#41", id=67, state=WAITING, blockCnt=2, waitCnt=44]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@920802f, ownerName=null, ownerId=
> -1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.jav
> a:2660)
> at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java
> :2705)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="Thread-9", id=66, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method)
> at sun.nio.ch.Iocp.access$300(Iocp.java:46)
> at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333)
> at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked synchronizers:
> java.util.concurrent.ThreadPoolExecutor$Worker@1849db1a
> Thread [name="Thread-8", id=65, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method)
> at sun.nio.ch.Iocp.access$300(Iocp.java:46)
> at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333)
> at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked synchronizers:
> java.util.concurrent.ThreadPoolExecutor$Worker@29a60c27
> Thread [name="Thread-7", id=64, state=RUNNABLE, blockCnt=1, waitCnt=0]
> at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method)
> at sun.nio.ch.Iocp.access$300(Iocp.java:46)
> at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333)
> at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked synchronizers:
> java.util.concurrent.ThreadPoolExecutor$Worker@79e18e38
> Thread [name="Thread-6", id=63, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method)
> at sun.nio.ch.Iocp.access$300(Iocp.java:46)
> at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333)
> at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked synchronizers:
> java.util.concurrent.ThreadPoolExecutor$Worker@3e2822
> Thread [name="Thread-5", id=62, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method)
> at sun.nio.ch.Iocp.access$300(Iocp.java:46)
> at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333)
> at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked synchronizers:
> java.util.concurrent.ThreadPoolExecutor$Worker@60fa3495
> Thread [name="Thread-4", id=61, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method)
> at sun.nio.ch.Iocp.access$300(Iocp.java:46)
> at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333)
> at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked synchronizers:
> java.util.concurrent.ThreadPoolExecutor$Worker@4602c2a9
> Thread [name="Thread-3", id=60, state=RUNNABLE, blockCnt=1, waitCnt=0]
> at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method)
> at sun.nio.ch.Iocp.access$300(Iocp.java:46)
> at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333)
> at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked synchronizers:
> java.util.concurrent.ThreadPoolExecutor$Worker@6a62689d
> Thread [name="Thread-2", id=59, state=RUNNABLE, blockCnt=1, waitCnt=0]
> at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method)
> at sun.nio.ch.Iocp.access$300(Iocp.java:46)
> at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333)
> at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked synchronizers:
> java.util.concurrent.ThreadPoolExecutor$Worker@4204541c
> Thread [name="Thread-1", id=58, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method)
> at sun.nio.ch.Iocp.access$300(Iocp.java:46)
> at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-data-loader-flusher-#40", id=57, state=WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@219ee97d, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2
> 039)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
> at o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-rest-3-#38", id=55, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@345eada4
> - locked java.util.Collections$UnmodifiableSet@475131ea
> - locked sun.nio.ch.WindowsSelectorImpl@4a7df0d7
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-rest-2-#37", id=54, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@1c7e3de4
> - locked java.util.Collections$UnmodifiableSet@5d8c8042
> - locked sun.nio.ch.WindowsSelectorImpl@131c2478
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-rest-1-#36", id=53, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@36c28584
> - locked java.util.Collections$UnmodifiableSet@5bbe1fa5
> - locked sun.nio.ch.WindowsSelectorImpl@62890a00
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-rest-0-#35", id=52, state=RUNNABLE, blockCnt=3, waitCnt=1]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@10c4cdcd
> - locked java.util.Collections$UnmodifiableSet@328894f4
> - locked sun.nio.ch.WindowsSelectorImpl@1960e637
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="nio-acceptor-tcp-rest-#39", id=56, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked sun.nio.ch.Util$3@33132611
> - locked java.util.Collections$UnmodifiableSet@50f2f1b
> - locked sun.nio.ch.WindowsSelectorImpl@7356247a
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2929)
> at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-client-listener-3-#32", id=49, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@6d993579
> - locked java.util.Collections$UnmodifiableSet@4710b575
> - locked sun.nio.ch.WindowsSelectorImpl@3b9b2d7b
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-client-listener-2-#31", id=48, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@79acab86
> - locked java.util.Collections$UnmodifiableSet@5d3a6b37
> - locked sun.nio.ch.WindowsSelectorImpl@1fc3d6f8
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-client-listener-1-#30", id=47, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7dd1c08e
> - locked java.util.Collections$UnmodifiableSet@613da429
> - locked sun.nio.ch.WindowsSelectorImpl@6f5018ea
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-client-listener-0-#29", id=46, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@6ea577ee
> - locked java.util.Collections$UnmodifiableSet@1d830f04
> - locked sun.nio.ch.WindowsSelectorImpl@1113fc71
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="nio-acceptor-client-listener-#33", id=50, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked sun.nio.ch.Util$3@7e6b31f1
> - locked java.util.Collections$UnmodifiableSet@3bb423d5
> - locked sun.nio.ch.WindowsSelectorImpl@3e4ae088
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2929)
> at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="tcp-comm-worker-#1", id=45, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@30b47344, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4287)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at o.a.i.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
> at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Thread [name="grid-nio-worker-tcp-comm-3-#27", id=43, state=RUNNABLE, blockCnt=1, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@1777381a
> - locked java.util.Collections$UnmodifiableSet@dc70baa
> - locked sun.nio.ch.WindowsSelectorImpl@6460f43a
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-comm-2-#26", id=42, state=RUNNABLE, blockCnt=3, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@775d85a1
> - locked java.util.Collections$UnmodifiableSet@59c4d4b8
> - locked sun.nio.ch.WindowsSelectorImpl@6ceb1e71
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-comm-1-#25", id=41, state=RUNNABLE, blockCnt=1, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@f49e6f2
> - locked java.util.Collections$UnmodifiableSet@14a20a37
> - locked sun.nio.ch.WindowsSelectorImpl@6bbc7d0f
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-nio-worker-tcp-comm-0-#24", id=40, state=RUNNABLE, blockCnt=2, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5f10fe9f
> - locked java.util.Collections$UnmodifiableSet@1faeedf9
> - locked sun.nio.ch.WindowsSelectorImpl@1b59af89
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151)
> at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="nio-acceptor-tcp-comm-#28", id=44, state=RUNNABLE, blockCnt=2, waitCnt=0]
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked sun.nio.ch.Util$3@1385fde8
> - locked java.util.Collections$UnmodifiableSet@4b390d01
> - locked sun.nio.ch.WindowsSelectorImpl@2390f0ba
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2929)
> at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="grid-timeout-worker-#23", id=39, state=TIMED_WAITING, blockCnt=3, waitCnt=164]
> Lock [object=java.lang.Object@756326a6, ownerName=null, ownerId=-1]
> at java.lang.Object.wait(Native Method)
> at o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:269)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="ignite-clock", id=38, state=TIMED_WAITING, blockCnt=0, waitCnt=5617]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.util.IgniteUtils$22.run(IgniteUtils.java:3379)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="pub-#19", id=34, state=TIMED_WAITING, blockCnt=1, waitCnt=2]
> Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33f5d231, ownerName=null, ownerId
> =-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j
> ava:2078)
> at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="jvm-pause-detector-worker", id=31, state=TIMED_WAITING, blockCnt=0, waitCnt=1360]
> at java.lang.Thread.sleep(Native Method)
> at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:100)
> Thread [name="data-streamer-stripe-7-#16", id=30, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-6-#15", id=29, state=WAITING, blockCnt=1, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-5-#14", id=28, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-4-#13", id=27, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-3-#12", id=26, state=WAITING, blockCnt=3, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-2-#11", id=25, state=WAITING, blockCnt=3, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-1-#10", id=24, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="data-streamer-stripe-0-#9", id=23, state=WAITING, blockCnt=3, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-7-#8", id=22, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-6-#7", id=21, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-5-#6", id=20, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-4-#5", id=19, state=WAITING, blockCnt=3, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-3-#4", id=18, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-2-#3", id=17, state=WAITING, blockCnt=3, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-1-#2", id=16, state=WAITING, blockCnt=0, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="sys-stripe-0-#1", id=15, state=WAITING, blockCnt=2, waitCnt=1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669)
> at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
> at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="RMI TCP Accept-0", id=14, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at java.net.DualStackPlainSocketImpl.accept0(Native Method)
> at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
> - locked java.net.SocksSocketImpl@49b5ff5f
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="RMI TCP Accept-49148", id=13, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at java.net.DualStackPlainSocketImpl.accept0(Native Method)
> at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
> - locked java.net.SocksSocketImpl@4ae7b26d
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="RMI TCP Accept-0", id=12, state=RUNNABLE, blockCnt=0, waitCnt=0]
> at java.net.DualStackPlainSocketImpl.accept0(Native Method)
> at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
> at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
> - locked java.net.SocksSocketImpl@6ed69eab
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
> at java.lang.Thread.run(Thread.java:748)
> Thread [name="Attach Listener", id=5, state=RUNNABLE, blockCnt=0, waitCnt=0]
> Thread [name="Signal Dispatcher", id=4, state=RUNNABLE, blockCnt=0, waitCnt=0]
> Thread [name="Finalizer", id=3, state=WAITING, blockCnt=193, waitCnt=5]
> Lock [object=java.lang.ref.ReferenceQueue$Lock@2ecc3c1, ownerName=null, ownerId=-1]
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
> Thread [name="Reference Handler", id=2, state=WAITING, blockCnt=6, waitCnt=4]
> Lock [object=java.lang.ref.Reference$Lock@64c59bc4, ownerName=null, ownerId=-1]
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> Thread [name="main", id=1, state=WAITING, blockCnt=5, waitCnt=5]
> Lock [object=java.util.concurrent.CountDownLatch$Sync@134629eb, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:8
> 36)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer
> .java:997)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.j
> ava:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at o.a.i.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:334)
> [18:05:37,502][SEVERE][sys-#62][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.NullPointerException]]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)