You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Naveen <na...@gmail.com> on 2018/11/10 15:03:24 UTC
Ways to improve re-balancing of partitions and how to monitor
re-balance progress
Hi
We are using Ignite 2.6
I have used rebalanceThreadPool to 16 and rebalanceBatchSize to 2 MB and we
have total data around 12 TB with 6 nodes cluster.
We are using very high end machines which are 128 CPU and 2 TB RAM etc, we
are finding ways to get re-balancing to be completed fast, right now with
the above configuration it takes very long time.
We have each cache loaded with 500M records, taking more than 24 hours to
re-balance single cache.
Its more than 3 days rebalancing has started, still its not yet over. I only
could completed (final) rebalancing log for only few caches around 4 to 5,
rest of the 40 caches not completed.
And, what I observed is, after removing a node from baseline and
re-balancing of data has started, there is no increase in the disk
utilization, I could only see RAM utilization has increased (from 200 GB to
500 GB). Does it mean that data gets re-balanced only in RAM and RAM updates
not getting reflected on DISK ??
Is there any better ways to monitor progress of re-balancing activity ?
And, also ways to speed up re-balance provided we have enough system
resources
Thanks
Naveen
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ways to improve re-balancing of partitions and how to monitor
re-balance progress
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Unfortunately if you have clients coming and parting from cluster all the
time, your rebalance can take long time.
There is a ticket on this issue:
https://issues.apache.org/jira/browse/IGNITE-7165
It looks like it will be fixed in 2.7.
Regards,
--
Ilya Kasnacheev
вт, 13 нояб. 2018 г. в 19:27, Naveen <na...@gmail.com>:
> Has any one got anything to say on this.
>
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ways to improve re-balancing of partitions and how to monitor
re-balance progress
Posted by Naveen <na...@gmail.com>.
Has any one got anything to say on this.
Thanks
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ways to improve re-balancing of partitions and how to monitor
re-balance progress
Posted by Naveen <na...@gmail.com>.
Add to above
I also see below, completed rebalancing and cancelled rebalancing logs after
any client joins connects to the cluster. Is it normal behavior of re
balance
[2018-11-08T20:06:36,936][INFO
][disco-event-worker-#341%EDIFCustomer%][GridDiscoveryManager] ^-- Node
[id=B5C2D0EE-72CB-49A6-848F-83DC7CBDCCD7, clusterState=ACTIVE]
[2018-11-08T20:06:36,936][INFO
][disco-event-worker-#341%EDIFCustomer%][GridDiscoveryManager] ^--
Baseline [id=1, size=5, online=5, offline=0]
[2018-11-08T20:06:36,936][INFO
][disco-event-worker-#341%EDIFCustomer%][GridDiscoveryManager] Data Regions
Configured:
[2018-11-08T20:06:36,936][INFO
][disco-event-worker-#341%EDIFCustomer%][GridDiscoveryManager] ^-- default
[initSize=256.0 MiB, maxSize=1.5 TiB, persistenceEnabled=true]
[2018-11-08T20:06:36,937][INFO ][exchange-worker-#342%EDIFCustomer%][time]
Started exchange init [topVer=AffinityTopologyVersion [topVer=23,
minorTopVer=0], crd=false, evt=NODE_JOINED,
evtNode=c3394ec2-a71a-4108-b720-0b2a10501fcc, customEvt=null,
allowMerge=true]
[2018-11-08T20:06:36,939][INFO
][exchange-worker-#342%EDIFCustomer%][GridDhtPartitionsExchangeFuture]
Finish exchange future [startVer=AffinityTopologyVersion [topVer=23,
minorTopVer=0], resVer=AffinityTopologyVersion [topVer=23, minorTopVer=0],
err=null]
[2018-11-08T20:06:36,939][INFO ][exchange-worker-#342%EDIFCustomer%][time]
Finished exchange init [topVer=AffinityTopologyVersion [topVer=23,
minorTopVer=0], crd=false]
[2018-11-08T20:06:36,949][INFO
][exchange-worker-#342%EDIFCustomer%][GridDhtPartitionDemander] Cancelled
rebalancing from all nodes [topology=AffinityTopologyVersion [topVer=22,
minorTopVer=1]]
[2018-11-08T20:06:36,949][INFO
][exchange-worker-#342%EDIFCustomer%][GridDhtPartitionDemander] Completed
rebalance future: RebalanceFuture [grp=CacheGroupContext
[grp=MapServCustCache], topVer=AffinityTopologyVersion [topVer=22,
minorTopVer=1], rebalanceId=1509]
[2018-11-08T20:06:36,949][INFO
][exchange-worker-#342%EDIFCustomer%][GridDhtPartitionDemander] Cancelled
rebalancing from all nodes [topology=AffinityTopologyVersion [topVer=22,
minorTopVer=1]]
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/