You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by swara <sw...@ekaplus.com> on 2020/10/12 06:15:01 UTC
java.lang.IllegalStateException: Getting affinity for too old
topology version that is already out of history
Hi
Getting this error run time.
ERROR IgniteInMemoryDAO:119 - [EVENT FAILURE Anonymous:null@unknown ->
/DefaultName/com.inmemory.IgniteInMemoryDAO] Exception :
java.lang.IllegalStateException: Getting affinity for too old topology
version that is already out of history [locNode=TcpDiscoveryNode
[id=cfb6e618-4a12-43db-b14a-68efdea05a8c,
consistentId=cfb6e618-4a12-43db-b14a-68efdea05a8c, addrs=ArrayList [],
sockAddrs=HashSet [], discPort=0, order=78, intOrder=0,
lastExchangeTime=1601918903890, loc=true, ver=2.8.1#20200521-sha1:86422096,
isClient=true], grp=SQL_PUBLIC_COLL_1602059401160,
topVer=AffinityTopologyVersion [topVer=78, minorTopVer=254],
lastAffChangeTopVer=AffinityTopologyVersion [topVer=78, minorTopVer=254],
head=AffinityTopologyVersion [topVer=78, minorTopVer=255],
history=[AffinityTopologyVersion [topVer=78, minorTopVer=255]]]
at
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:817)
at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.assignment(GridCacheAffinityManager.java:255)
at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.assignment(GridCacheAffinityManager.java:237)
at
org.apache.ignite.internal.processors.query.h2.twostep.ReducePartitionMapper.stableDataNodesMap(ReducePartitionMapper.java:268)
at
org.apache.ignite.internal.processors.query.h2.twostep.ReducePartitionMapper.stableDataNodes(ReducePartitionMapper.java:212)
Thank You
Swara Yerukola.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: java.lang.IllegalStateException: Getting affinity for too old
topology version that is already out of history
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Why not, if it fixes the issue for you. Then you can try 2.9.0 once it is
out.
Unfortunately, a small snippet of server log is not sufficient, complete
log is needed. I can already spot some communication problems, though.
Regards,
--
Ilya Kasnacheev
чт, 15 окт. 2020 г. в 09:50, swara <sw...@ekaplus.com>:
> Hello
>
> We are using this type of cache design from last 3 years with 2.5.6
> version.
> we never got any issue.
> When we upgraded to 2.8.1 we are getting this type of errors.
> Should we downgrade to 2.8.0 or 2.5.6. Please suggest?
>
> Thank You
> Swara
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: java.lang.IllegalStateException: Getting affinity for too old
topology version that is already out of history
Posted by swara <sw...@ekaplus.com>.
Hello
We are using this type of cache design from last 3 years with 2.5.6 version.
we never got any issue.
When we upgraded to 2.8.1 we are getting this type of errors.
Should we downgrade to 2.8.0 or 2.5.6. Please suggest?
Thank You
Swara
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: java.lang.IllegalStateException: Getting affinity for too old
topology version that is already out of history
Posted by swara <sw...@ekaplus.com>.
Hello
Same error we got again today.
Ignite.log-
java.lang.IllegalStateException: Getting affinity for too old topology
version that is already out of history [locNode=TcpDiscoveryNode
[id=57473290-f181-49e5-9f5e-81188abc012f,
consistentId=57473290-f181-49e5-9f5e-81188abc012f, addrs=ArrayList [
127.0.0.1,], sockAddrs=HashSet [ /127.0.0.1:0,], discPort=0, order=2,
intOrder=0, lastExchangeTime=1602615417319, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=true],
grp=SQL_PUBLIC_COLL_1602660610976, topVer=AffinityTopologyVersion [topVer=2,
minorTopVer=129], lastAffChangeTopVer=AffinityTopologyVersion [topVer=2,
minorTopVer=129], head=AffinityTopologyVersion [topVer=2, minorTopVer=130],
history=[AffinityTopologyVersion [topVer=2, minorTopVer=130]]]
at
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:817)
at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.assignment(GridCacheAffinityManager.java:255)
at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.assignment(GridCacheAffinityManager.java:237)
Server log-
[07:29:49,967][INFO][db-checkpoint-thread-#60][GridCacheDatabaseSharedManager]
Skipping checkpoint (no pages were modified)
[checkpointBeforeLockTime=126ms, checkpointLockWait=0ms,
checkpointListenersExecuteTime=118ms, checkpointLockHoldTime=130ms,
reason='timeout']
[07:30:10,991][INFO][exchange-worker-#46][time] Started exchange init
[topVer=AffinityTopologyVersion [topVer=2, minorTopVer=130], crd=true,
evt=DISCOVERY_CUSTOM_EVT, evtNode=57473290-f181-49e5-9f5e-81188abc012f,
customEvt=DynamicCacheChangeBatch
[id=057f3532571-1385aa3f-7385-4236-8361-193928f265ec, reqs=ArrayList
[DynamicCacheChangeRequest [cacheName=SQL_PUBLIC_COLL_1602660610976,
hasCfg=true, nodeId=57473290-f181-49e5-9f5e-81188abc012f,
clientStartOnly=false, stop=false, destroy=false, disabledAfterStartfalse]],
exchangeActions=ExchangeActions
[startCaches=[SQL_PUBLIC_COLL_1602660610976], stopCaches=null,
startGrps=[SQL_PUBLIC_COLL_1602660610976], stopGrps=[], resetParts=null,
stateChangeRequest=null], startCaches=false], allowMerge=false,
exchangeFreeSwitch=false]
[07:30:11,019][INFO][exchange-worker-#46][GridCacheProcessor] Started cache
[name=SQL_PUBLIC_COLL_1602660610976, id=238456958, dataRegionName=default,
mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false]
[07:30:11,042][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Finished waiting for partition release future
[topVer=AffinityTopologyVersion [topVer=2, minorTopVer=130], waitTime=0ms,
futInfo=NA, mode=DISTRIBUTED]
[07:30:11,042][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Finished waiting for partitions release latch: ServerLatch [permits=0,
pendingAcks=HashSet [], super=CompletableLatch [id=CompletableLatchUid
[id=exchange, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=130]]]]
[07:30:11,042][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Finished waiting for partition release future
[topVer=AffinityTopologyVersion [topVer=2, minorTopVer=130], waitTime=0ms,
futInfo=NA, mode=LOCAL]
[07:30:11,132][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], resVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130]]
[07:30:11,240][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Finish exchange future [startVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], resVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], err=null, rebalanced=true, wasRebalanced=true]
[07:30:11,257][INFO][exchange-worker-#46][GridCacheProcessor] Finish proxy
initialization, cacheName=SQL_PUBLIC_COLL_1602660610976,
localNodeId=bea04023-2512-4900-b9f0-ea363f7ef66f
[07:30:11,258][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Completed partition exchange
[localNode=bea04023-2512-4900-b9f0-ea363f7ef66f,
exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
[topVer=2, minorTopVer=130], evt=DISCOVERY_CUSTOM_EVT,
evtNode=TcpDiscoveryNode [id=57473290-f181-49e5-9f5e-81188abc012f,
consistentId=57473290-f181-49e5-9f5e-81188abc012f, addrs=ArrayList [],
sockAddrs=HashSet [], discPort=0, order=2, intOrder=2,
lastExchangeTime=1602615417390, loc=false, ver=2.8.1#20200521-sha1:86422096,
isClient=true], done=true, newCrdFut=null], topVer=AffinityTopologyVersion
[topVer=2, minorTopVer=130]]
[07:30:11,258][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Exchange timings [startVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], resVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], stage="Waiting in exchange queue" (0 ms), stage="Exchange
parameters initialization" (0 ms), stage="Update caches registry" (1 ms),
stage="Start caches" (27 ms), stage="Affinity initialization on cache group
start" (0 ms), stage="Determine exchange type" (0 ms), stage="Preloading
notification" (0 ms), stage="WAL history reservation" (21 ms), stage="Wait
partitions release" (0 ms), stage="Wait partitions release latch" (0 ms),
stage="Wait partitions release" (0 ms), stage="Restore partition states" (4
ms), stage="After states restored callback" (85 ms), stage="Waiting for all
single messages" (0 ms), stage="Affinity recalculation (crd)" (0 ms),
stage="Collect update counters and create affinity messages" (0 ms),
stage="Validate partitions states" (8 ms), stage="Assign partitions states"
(23 ms), stage="Apply update counters" (26 ms), stage="Full message
preparing" (49 ms), stage="Full message sending" (0 ms), stage="Exchange
done" (17 ms), stage="Total time" (261 ms)]
[07:30:11,258][INFO][exchange-worker-#46][GridDhtPartitionsExchangeFuture]
Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], resVer=AffinityTopologyVersion [topVer=2,
minorTopVer=130], stage="Affinity initialization on cache group start
[grp=SQL_PUBLIC_COLL_1602660610976]" (0 ms) (parent=Affinity initialization
on cache group start), stage="Restore partition states
[grp=SQL_PUBLIC_COLL_1602660610976]" (4 ms) (parent=Restore partition
states), stage="Restore partition states [grp=ignite-sys-cache]" (0 ms)
(parent=Restore partition states), stage="Restore partition states
[grp=SQL_PUBLIC_COLL_1602660408621]" (0 ms) (parent=Restore partition
states)]
[07:30:11,258][INFO][exchange-worker-#46][time] Finished exchange init
[topVer=AffinityTopologyVersion [topVer=2, minorTopVer=130], crd=true]
[07:30:11,308][INFO][exchange-worker-#46][GridCachePartitionExchangeManager]
Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion
[topVer=2, minorTopVer=130], force=false, evt=DISCOVERY_CUSTOM_EVT,
node=57473290-f181-49e5-9f5e-81188abc012f]
[07:30:13,301][WARNING][grid-nio-worker-tcp-comm-0-#24][TcpCommunicationSpi]
Communication SPI session write timed out (consider increasing
'socketWriteTimeout' configuration property) [remoteAddr=/,
writeTimeout=2000]
[07:30:16,561][INFO][grid-nio-worker-tcp-comm-1-#25][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/, rmtAddr=/]
[07:30:16,562][INFO][grid-nio-worker-tcp-comm-1-#25][TcpCommunicationSpi]
Received incoming connection from remote node while connecting to this node,
rejecting [locNode=bea04023-2512-4900-b9f0-ea363f7ef66f, locNodeOrder=1,
rmtNode=57473290-f181-49e5-9f5e-81188abc012f, rmtNodeOrder=2]
[07:30:16,764][INFO][grid-nio-worker-tcp-comm-2-#26][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/, rmtAddr=/]
[07:30:16,764][INFO][grid-nio-worker-tcp-comm-2-#26][TcpCommunicationSpi]
Received incoming connection from remote node while connecting to this node,
rejecting [locNode=bea04023-2512-4900-b9f0-ea363f7ef66f, locNodeOrder=1,
rmtNode=57473290-f181-49e5-9f5e-81188abc012f, rmtNodeOrder=2]
Thank You
Swara
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: java.lang.IllegalStateException: Getting affinity for too old
topology version that is already out of history
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
How often do you create or drop tables? Minor version of 255 suggests a lot
of operations.
Can you share logs for your nodes prior to that error?
Regards,
--
Ilya Kasnacheev
пн, 12 окт. 2020 г. в 16:04, swara <sw...@ekaplus.com>:
> one per table, whenever the table definition/data changes
> hourly/daily/weekly.
>
> please suggest if there is any other way to do joins without using cache?
>
> Thank You
> Swara
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: java.lang.IllegalStateException: Getting affinity for too old
topology version that is already out of history
Posted by swara <sw...@ekaplus.com>.
one per table, whenever the table definition/data changes
hourly/daily/weekly.
please suggest if there is any other way to do joins without using cache?
Thank You
Swara
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: java.lang.IllegalStateException: Getting affinity for too old
topology version that is already out of history
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
How many caches do you have? How often are they created?
Regards,
--
Ilya Kasnacheev
пн, 12 окт. 2020 г. в 13:35, swara <sw...@ekaplus.com>:
> We are creating caches for each table and joining tables.
>
> Thank You
> Swara
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: java.lang.IllegalStateException: Getting affinity for too old
topology version that is already out of history
Posted by swara <sw...@ekaplus.com>.
We are creating caches for each table and joining tables.
Thank You
Swara
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: java.lang.IllegalStateException: Getting affinity for too old
topology version that is already out of history
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Looks like you have an SQL query which executes for ages and ran out of
topology history.
That, or some other weird issue (you have minorTopVer of 250+ which is
unusual). Can you provide logs for your server nodes?
Do you have any short-lived caches which are created and destroyed
frequently? I advise against it, caches should be long-lived for optimal
performance and decreased risk of running into issues like this one.
Regards,
--
Ilya Kasnacheev
пн, 12 окт. 2020 г. в 09:15, swara <sw...@ekaplus.com>:
> Hi
>
> Getting this error run time.
>
>
> ERROR IgniteInMemoryDAO:119 - [EVENT FAILURE Anonymous:null@unknown ->
> /DefaultName/com.inmemory.IgniteInMemoryDAO] Exception :
> java.lang.IllegalStateException: Getting affinity for too old topology
> version that is already out of history [locNode=TcpDiscoveryNode
> [id=cfb6e618-4a12-43db-b14a-68efdea05a8c,
> consistentId=cfb6e618-4a12-43db-b14a-68efdea05a8c, addrs=ArrayList [],
> sockAddrs=HashSet [], discPort=0, order=78, intOrder=0,
> lastExchangeTime=1601918903890, loc=true, ver=2.8.1#20200521-sha1:86422096,
> isClient=true], grp=SQL_PUBLIC_COLL_1602059401160,
> topVer=AffinityTopologyVersion [topVer=78, minorTopVer=254],
> lastAffChangeTopVer=AffinityTopologyVersion [topVer=78, minorTopVer=254],
> head=AffinityTopologyVersion [topVer=78, minorTopVer=255],
> history=[AffinityTopologyVersion [topVer=78, minorTopVer=255]]]
> at
>
> org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:817)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.assignment(GridCacheAffinityManager.java:255)
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.assignment(GridCacheAffinityManager.java:237)
> at
>
> org.apache.ignite.internal.processors.query.h2.twostep.ReducePartitionMapper.stableDataNodesMap(ReducePartitionMapper.java:268)
> at
>
> org.apache.ignite.internal.processors.query.h2.twostep.ReducePartitionMapper.stableDataNodes(ReducePartitionMapper.java:212)
>
>
> Thank You
> Swara Yerukola.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>