You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by 18624049226 <18...@163.com> on 2020/05/03 01:48:52 UTC
Group eviction in progress for a very long time
Hi community,
Two nodes are added to a running cluster with persistence enabled, after
several days, it seems that the rebalancing is not over. The log output
the following information about every 2 minutes:
why?How to end the rebalancing?
[2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
Eviction in progress [permits=0, threads=4, groups=4,
remainingPartsToEvict=293]
[2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812,
remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
[2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728,
remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
[2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
Group eviction in progress [grpName=CO_CO, grpId=64322847,
remainingPartsToEvict=114, partsEvictInProgress=6, totalParts=544]
[2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
Group eviction in progress [grpName=CO_CUST, grpId=1684722246,
remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
[2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
Partitions have been scheduled for eviction: [grpId=64322847,
grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601,
669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
[2020-04-30T21:58:32,288][INFO][grid-timeout-worker-#39][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency'to 0)
^-- Node [id=9ebed61d, uptime=4days, 19:34:36.448]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=18%, avg=18.24%, GC=0%]
^-- PageMemory [pages=36986314]
^-- Heap [used=12815MB, free=37.43%, comm=20480MB]
^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=146170MB, free=28.63%, comm=204800MB]
^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=134048MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=134047MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=16, idle=0, qSize=3]
^-- Striped thread pool [active=0, idle=16, qSize=0]
[2020-04-30T21:58:49,329][INFO][db-checkpoint-thread-#409][GridCacheDatabaseSharedManager]
Skipping checkpoint (no pages were modified)
[checkpointBeforeLockTime=13ms, checkpointLockWait=0ms,
checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=7ms,
reason='timeout']
[2020-04-30T21:59:32,297][INFO][grid-timeout-worker-#39][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency'to 0)
^-- Node [id=9ebed61d, uptime=4days, 19:35:36.457]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=17.9%, avg=18.24%, GC=0%]
^-- PageMemory [pages=36986314]
^-- Heap [used=8335MB, free=59.3%, comm=20480MB]
^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
^-- default region [used=146170MB, free=28.63%, comm=204800MB]
^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=134048MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=134047MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=6, idle=10, qSize=0]
^-- Striped thread pool [active=0, idle=16, qSize=0]
[2020-04-30T21:59:49,898][INFO][sys-#28325][PartitionsEvictManager]
Eviction in progress [permits=1, threads=4, groups=4,
remainingPartsToEvict=291]
[2020-04-30T21:59:49,898][INFO][sys-#28325][PartitionsEvictManager]
Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812,
remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
[2020-04-30T21:59:49,898][INFO][sys-#28325][PartitionsEvictManager]
Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728,
remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
[2020-04-30T21:59:49,898][INFO][sys-#28325][PartitionsEvictManager]
Group eviction in progress [grpName=CO_CO, grpId=64322847,
remainingPartsToEvict=112, partsEvictInProgress=6, totalParts=544]
[2020-04-30T21:59:49,899][INFO][sys-#28325][PartitionsEvictManager]
Group eviction in progress [grpName=CO_CUST, grpId=1684722246,
remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
[2020-04-30T21:59:49,899][INFO][sys-#28325][PartitionsEvictManager]
Partitions have been scheduled for eviction: [grpId=64322847,
grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601,
669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
[2020-04-30T22:00:32,302][INFO][grid-timeout-worker-#39][IgniteKernal]
Re: Group eviction in progress for a very long time
Posted by "henrique.arroyo" <he...@ifood.com.br>.
This is related to the following bug
https://issues.apache.org/jira/browse/IGNITE-11974
We are currently running version 2.8.1 and it is affecting our servers as
well, the results are disastrous.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Group eviction in progress for a very long time
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Can you search the previous logs for any exceptions coming from Ignite
internals? Maybe something did indeed break down.
Regards,
--
Ilya Kasnacheev
пн, 4 мая 2020 г. в 16:22, 18624049226 <18...@163.com>:
> Hi Ilya,
>
> It's hard to say why rebalancing is not over yet, judging from this log
> portion alone, but "Group eviction in progress" would probably mean that
> some partitions were successfully rebalanced and they may now be removed
> from "old" backup node.
>
> That's how I understand it,According to the visor, there are more backups
> than the primary ones.
>
>
> Are you getting the same #'s of partitions every time, or do they change?
> Any irregularities in your logs other than this one?
>
> At present, the cluster is in the state of no access, so the current log
> scrolls out the following information about every 2 seconds, basically the
> same.
>
> We have 5 nodes, 2 of which have the following logs.
>
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вс, 3 мая 2020 г. в 04:49, 18624049226 <18...@163.com>:
>
>> Hi community,
>>
>> Two nodes are added to a running cluster with persistence enabled, after
>> several days, it seems that the rebalancing is not over. The log output the
>> following information about every 2 minutes:
>>
>> why?How to end the rebalancing?
>> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
>> Eviction in progress [permits=0, threads=4, groups=4,
>> remainingPartsToEvict=293]
>> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
>> Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812,
>> remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
>> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
>> Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728,
>> remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
>> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
>> Group eviction in progress [grpName=CO_CO, grpId=64322847,
>> remainingPartsToEvict=114, partsEvictInProgress=6, totalParts=544]
>> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
>> Group eviction in progress [grpName=CO_CUST, grpId=1684722246,
>> remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
>> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
>> Partitions have been scheduled for eviction: [grpId=64322847,
>> grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601,
>> 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
>> [2020-04-30T21:58:32,288][INFO ][grid-timeout-worker-#39][IgniteKernal]
>> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>> ^-- Node [id=9ebed61d, uptime=4 days, 19:34:36.448]
>> ^-- H/N/C [hosts=5, nodes=5, CPUs=80]
>> ^-- CPU [cur=18%, avg=18.24%, GC=0%]
>> ^-- PageMemory [pages=36986314]
>> ^-- Heap [used=12815MB, free=37.43%, comm=20480MB]
>> ^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
>> ^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
>> ^-- default region [used=146170MB, free=28.63%, comm=204800MB]
>> ^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
>> ^-- TxLog region [used=0MB, free=100%, comm=100MB]
>> ^-- Ignite persistence [used=134048MB]
>> ^-- sysMemPlc region [used=0MB]
>> ^-- default region [used=134047MB]
>> ^-- metastoreMemPlc region [used=0MB]
>> ^-- TxLog region [used=0MB]
>> ^-- Outbound messages queue [size=0]
>> ^-- Public thread pool [active=0, idle=0, qSize=0]
>> ^-- System thread pool [active=16, idle=0, qSize=3]
>> ^-- Striped thread pool [active=0, idle=16, qSize=0]
>> [2020-04-30T21:58:49,329][INFO ][db-checkpoint-thread-#409][GridCacheDatabaseSharedManager]
>> Skipping checkpoint (no pages were modified)
>> [checkpointBeforeLockTime=13ms, checkpointLockWait=0ms,
>> checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=7ms, reason=
>> 'timeout']
>> [2020-04-30T21:59:32,297][INFO ][grid-timeout-worker-#39][IgniteKernal]
>> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>> ^-- Node [id=9ebed61d, uptime=4 days, 19:35:36.457]
>> ^-- H/N/C [hosts=5, nodes=5, CPUs=80]
>> ^-- CPU [cur=17.9%, avg=18.24%, GC=0%]
>> ^-- PageMemory [pages=36986314]
>> ^-- Heap [used=8335MB, free=59.3%, comm=20480MB]
>> ^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
>> ^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
>> ^-- default region [used=146170MB, free=28.63%, comm=204800MB]
>> ^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
>> ^-- TxLog region [used=0MB, free=100%, comm=100MB]
>> ^-- Ignite persistence [used=134048MB]
>> ^-- sysMemPlc region [used=0MB]
>> ^-- default region [used=134047MB]
>> ^-- metastoreMemPlc region [used=0MB]
>> ^-- TxLog region [used=0MB]
>> ^-- Outbound messages queue [size=0]
>> ^-- Public thread pool [active=0, idle=0, qSize=0]
>> ^-- System thread pool [active=6, idle=10, qSize=0]
>> ^-- Striped thread pool [active=0, idle=16, qSize=0]
>> [2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager]
>> Eviction in progress [permits=1, threads=4, groups=4,
>> remainingPartsToEvict=291]
>> [2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager]
>> Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812,
>> remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
>> [2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager]
>> Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728,
>> remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
>> [2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager]
>> Group eviction in progress [grpName=CO_CO, grpId=64322847,
>> remainingPartsToEvict=112, partsEvictInProgress=6, totalParts=544]
>> [2020-04-30T21:59:49,899][INFO ][sys-#28325][PartitionsEvictManager]
>> Group eviction in progress [grpName=CO_CUST, grpId=1684722246,
>> remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
>> [2020-04-30T21:59:49,899][INFO ][sys-#28325][PartitionsEvictManager]
>> Partitions have been scheduled for eviction: [grpId=64322847,
>> grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601,
>> 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
>> [2020-04-30T22:00:32,302][INFO ][grid-timeout-worker-#39][IgniteKernal]
>>
>>
>>
Re: Group eviction in progress for a very long time
Posted by 18624049226 <18...@163.com>.
Hi Ilya,
> It's hard to say why rebalancing is not over yet, judging from this
> log portion alone, but "Group eviction in progress" would probably
> mean that some partitions were successfully rebalanced and they may
> now be removed from "old" backup node.
That's how I understand it,According to the visor, there are more
backups than the primary ones.
>
> Are you getting the same #'s of partitions every time, or do they
> change? Any irregularities in your logs other than this one?
At present, the cluster is in the state of no access, so the current log
scrolls out the following information about every 2 seconds, basically
the same.
We have 5 nodes, 2 of which have the following logs.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вс, 3 мая 2020 г. в 04:49, 18624049226 <18624049226@163.com
> <ma...@163.com>>:
>
> Hi community,
>
> Two nodes are added to a running cluster with persistence enabled,
> after several days, it seems that the rebalancing is not over. The
> log output the following information about every 2 minutes:
>
> why?How to end the rebalancing?
>
> [2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
> Eviction in progress [permits=0, threads=4, groups=4,
> remainingPartsToEvict=293]
> [2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812,
> remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
> [2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
> Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728,
> remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
> [2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CO, grpId=64322847,
> remainingPartsToEvict=114, partsEvictInProgress=6, totalParts=544]
> [2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CUST, grpId=1684722246,
> remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
> [2020-04-30T21:57:49,898][INFO][sys-#28326][PartitionsEvictManager]
> Partitions have been scheduled for eviction: [grpId=64322847,
> grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583,
> 591, 601, 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
> [2020-04-30T21:58:32,288][INFO][grid-timeout-worker-#39][IgniteKernal]
>
> Metrics for local node (to disable set 'metricsLogFrequency'to 0)
> ^-- Node [id=9ebed61d, uptime=4days, 19:34:36.448]
> ^-- H/N/C [hosts=5, nodes=5, CPUs=80]
> ^-- CPU [cur=18%, avg=18.24%, GC=0%]
> ^-- PageMemory [pages=36986314]
> ^-- Heap [used=12815MB, free=37.43%, comm=20480MB]
> ^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
> ^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
> ^-- default region [used=146170MB, free=28.63%, comm=204800MB]
> ^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
> ^-- TxLog region [used=0MB, free=100%, comm=100MB]
> ^-- Ignite persistence [used=134048MB]
> ^-- sysMemPlc region [used=0MB]
> ^-- default region [used=134047MB]
> ^-- metastoreMemPlc region [used=0MB]
> ^-- TxLog region [used=0MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=16, idle=0, qSize=3]
> ^-- Striped thread pool [active=0, idle=16, qSize=0]
> [2020-04-30T21:58:49,329][INFO][db-checkpoint-thread-#409][GridCacheDatabaseSharedManager]
> Skipping checkpoint (no pages were modified)
> [checkpointBeforeLockTime=13ms, checkpointLockWait=0ms,
> checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=7ms,
> reason='timeout']
> [2020-04-30T21:59:32,297][INFO][grid-timeout-worker-#39][IgniteKernal]
>
> Metrics for local node (to disable set 'metricsLogFrequency'to 0)
> ^-- Node [id=9ebed61d, uptime=4days, 19:35:36.457]
> ^-- H/N/C [hosts=5, nodes=5, CPUs=80]
> ^-- CPU [cur=17.9%, avg=18.24%, GC=0%]
> ^-- PageMemory [pages=36986314]
> ^-- Heap [used=8335MB, free=59.3%, comm=20480MB]
> ^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
> ^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
> ^-- default region [used=146170MB, free=28.63%, comm=204800MB]
> ^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
> ^-- TxLog region [used=0MB, free=100%, comm=100MB]
> ^-- Ignite persistence [used=134048MB]
> ^-- sysMemPlc region [used=0MB]
> ^-- default region [used=134047MB]
> ^-- metastoreMemPlc region [used=0MB]
> ^-- TxLog region [used=0MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=6, idle=10, qSize=0]
> ^-- Striped thread pool [active=0, idle=16, qSize=0]
> [2020-04-30T21:59:49,898][INFO][sys-#28325][PartitionsEvictManager]
> Eviction in progress [permits=1, threads=4, groups=4,
> remainingPartsToEvict=291]
> [2020-04-30T21:59:49,898][INFO][sys-#28325][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812,
> remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
> [2020-04-30T21:59:49,898][INFO][sys-#28325][PartitionsEvictManager]
> Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728,
> remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
> [2020-04-30T21:59:49,898][INFO][sys-#28325][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CO, grpId=64322847,
> remainingPartsToEvict=112, partsEvictInProgress=6, totalParts=544]
> [2020-04-30T21:59:49,899][INFO][sys-#28325][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CUST, grpId=1684722246,
> remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
> [2020-04-30T21:59:49,899][INFO][sys-#28325][PartitionsEvictManager]
> Partitions have been scheduled for eviction: [grpId=64322847,
> grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583,
> 591, 601, 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
> [2020-04-30T22:00:32,302][INFO][grid-timeout-worker-#39][IgniteKernal]
>
>
>
Re: Group eviction in progress for a very long time
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
It's hard to say why rebalancing is not over yet, judging from this log
portion alone, but "Group eviction in progress" would probably mean that
some partitions were successfully rebalanced and they may now be removed
from "old" backup node.
Are you getting the same #'s of partitions every time, or do they change?
Any irregularities in your logs other than this one?
Regards,
--
Ilya Kasnacheev
вс, 3 мая 2020 г. в 04:49, 18624049226 <18...@163.com>:
> Hi community,
>
> Two nodes are added to a running cluster with persistence enabled, after
> several days, it seems that the rebalancing is not over. The log output the
> following information about every 2 minutes:
>
> why?How to end the rebalancing?
> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
> Eviction in progress [permits=0, threads=4, groups=4,
> remainingPartsToEvict=293]
> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812,
> remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
> Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728,
> remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CO, grpId=64322847,
> remainingPartsToEvict=114, partsEvictInProgress=6, totalParts=544]
> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CUST, grpId=1684722246,
> remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
> [2020-04-30T21:57:49,898][INFO ][sys-#28326][PartitionsEvictManager]
> Partitions have been scheduled for eviction: [grpId=64322847,
> grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601,
> 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
> [2020-04-30T21:58:32,288][INFO ][grid-timeout-worker-#39][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=9ebed61d, uptime=4 days, 19:34:36.448]
> ^-- H/N/C [hosts=5, nodes=5, CPUs=80]
> ^-- CPU [cur=18%, avg=18.24%, GC=0%]
> ^-- PageMemory [pages=36986314]
> ^-- Heap [used=12815MB, free=37.43%, comm=20480MB]
> ^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
> ^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
> ^-- default region [used=146170MB, free=28.63%, comm=204800MB]
> ^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
> ^-- TxLog region [used=0MB, free=100%, comm=100MB]
> ^-- Ignite persistence [used=134048MB]
> ^-- sysMemPlc region [used=0MB]
> ^-- default region [used=134047MB]
> ^-- metastoreMemPlc region [used=0MB]
> ^-- TxLog region [used=0MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=16, idle=0, qSize=3]
> ^-- Striped thread pool [active=0, idle=16, qSize=0]
> [2020-04-30T21:58:49,329][INFO ][db-checkpoint-thread-#409][GridCacheDatabaseSharedManager]
> Skipping checkpoint (no pages were modified)
> [checkpointBeforeLockTime=13ms, checkpointLockWait=0ms,
> checkpointListenersExecuteTime=6ms, checkpointLockHoldTime=7ms, reason=
> 'timeout']
> [2020-04-30T21:59:32,297][INFO ][grid-timeout-worker-#39][IgniteKernal]
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=9ebed61d, uptime=4 days, 19:35:36.457]
> ^-- H/N/C [hosts=5, nodes=5, CPUs=80]
> ^-- CPU [cur=17.9%, avg=18.24%, GC=0%]
> ^-- PageMemory [pages=36986314]
> ^-- Heap [used=8335MB, free=59.3%, comm=20480MB]
> ^-- Off-heap [used=146170MB, free=28.73%, comm=205000MB]
> ^-- sysMemPlc region [used=0MB, free=99.99%, comm=100MB]
> ^-- default region [used=146170MB, free=28.63%, comm=204800MB]
> ^-- metastoreMemPlc region [used=0MB, free=99.84%, comm=0MB]
> ^-- TxLog region [used=0MB, free=100%, comm=100MB]
> ^-- Ignite persistence [used=134048MB]
> ^-- sysMemPlc region [used=0MB]
> ^-- default region [used=134047MB]
> ^-- metastoreMemPlc region [used=0MB]
> ^-- TxLog region [used=0MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=6, idle=10, qSize=0]
> ^-- Striped thread pool [active=0, idle=16, qSize=0]
> [2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager]
> Eviction in progress [permits=1, threads=4, groups=4,
> remainingPartsToEvict=291]
> [2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CO_LINE, grpId=-1588248812,
> remainingPartsToEvict=115, partsEvictInProgress=0, totalParts=538]
> [2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager]
> Group eviction in progress [grpName=PI_COM_DAY, grpId=-1904194728,
> remainingPartsToEvict=25, partsEvictInProgress=0, totalParts=448]
> [2020-04-30T21:59:49,898][INFO ][sys-#28325][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CO, grpId=64322847,
> remainingPartsToEvict=112, partsEvictInProgress=6, totalParts=544]
> [2020-04-30T21:59:49,899][INFO ][sys-#28325][PartitionsEvictManager]
> Group eviction in progress [grpName=CO_CUST, grpId=1684722246,
> remainingPartsToEvict=39, partsEvictInProgress=0, totalParts=462]
> [2020-04-30T21:59:49,899][INFO ][sys-#28325][PartitionsEvictManager]
> Partitions have been scheduled for eviction: [grpId=64322847,
> grpName=CO_CO, eviction=[40, 58, 292, 305, 324, 362, 551, 583, 591, 601,
> 669, 701, 726, 741-742, 785, 821, 855, 922, 956, 1009]]
> [2020-04-30T22:00:32,302][INFO ][grid-timeout-worker-#39][IgniteKernal]
>
>
>