You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Sergii Tyshlek <st...@llnw.com> on 2016/08/16 09:54:10 UTC
Rebalance is skipped
Hello there!
Some time ago we started moving from old GridGain to current Apache Ignite
(1.6, now 1.7).
Here are some cache config properties we use:
--------------------------------------------
cacheMode=PARTITIONED
atomicityMode=ATOMIC
atomicWriteOrderMode=PRIMARY
writeSynchronizationMode=PRIMARY_SYNC
rebalanceMode=ASYNC
rebalanceBatchesPrefetchCount=2
rebalanceDelay=30000
rebalanceTimeout=10000
backups=1
affinity=FairAffinityFunction
affinity.partitions=1024
--------------------------------------------
At first, everything looked OK, but then I noticed that our data is not
distributed evenly between nodes (despite the fact we use
FairAffinityFunction, coordinator node hoards most of the cache entries).
Later I discovered (through wrapping my custom class around
FairAffinityFunction) that it works as expected, but the rebalancing is not.
Short after starting 6 nodes (after the last one joins the topology), such
debug logs appear:
---------------------------------------------------------
2016-08-15 07:36:43,070 DEBUG [exchange-worker-#318%EQGrid%]
[preloader.GridDhtPreloader] - <p_queryResults> Skipping partition
assignment (state is not MOVING): GridDhtLocalPartition [id=0,
map=org.apache.ignite.internal.processors.cache.GridCacheCon
currentMapImpl@5c35248d, rmvQueue=GridCircularBuffer [sizeMask=255,
idxGen=0], cntr=0, state=OWNING, reservations=0, empty=true,
createTime=08/15/2016 07:36:33]
...
// repeats total of 1024 times, where id=0..1023, one for every partition
// then it's followed by
2016-08-15 07:36:43,177 DEBUG [exchange-worker-#318%EQGrid%]
[preloader.GridDhtPartitionDemander] - <p_queryResults> Adding partition
assignments: GridDhtPreloaderAssignments [topVer=AffinityTopologyVersion
[topVer=6, minorTopVer=0], cancelled=false, exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], nodeId=383991fb,
evt=NODE_JOINED], super={}]
2016-08-15 07:36:43,177 DEBUG [exchange-worker-#318%EQGrid%]
[preloader.GridDhtPartitionDemander] - <p_queryResults> Rebalancing is not
required [cache=p_queryResults, topology=AffinityTopologyVersion [topVer=6,
minorTopVer=0]]
2016-08-15 07:36:43,178 DEBUG [exchange-worker-#318%EQGrid%]
[preloader.GridDhtPartitionDemander] - <p_queryResults> Completed rebalance
future: RebalanceFuture [sndStoppedEvnt=false,
topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], updateSeq=6]
2016-08-15 07:36:43,179 INFO [exchange-worker-#318%EQGrid%]
[cache.GridCachePartitionExchangeManager] - Skipping rebalancing (nothing
scheduled) [top=AffinityTopologyVersion [topVer=6, minorTopVer=0],
evt=NODE_JOINED, node=383991fb-5453-4893-9040-1baa1291881a]
---------------------------------------------------------
So I started digging. Using GridDhtPartitionTopology, I got partitions map,
which (aggregated) looked like this:
---------------------------------------------------------
Node: 38ae4165-474d-4ed4-a292-cca78b8df5c3, partitions: {MOVING=340}
Node: 8ac8d327-dc59-473f-a3e1-c5861f63f0e6, partitions: {MOVING=341}
Node: c7047158-9e7b-494f-bceb-3a5774853a6c, partitions: {MOVING=342}
Node: c9cc1a1f-f037-43c8-8855-0f1ccb8f0ec5, partitions: {MOVING=342}
Node: dce874ff-cc1e-41c8-9e82-abfb3dfa535e, partitions: {OWNING=1024}
Node: de783f6d-dc48-46b8-a387-91dd3d181150, partitions: {MOVING=342}
---------------------------------------------------------
Important point is that such distribution never changes, neither right
after grid start, nor after few hours. Ingesting (or not ingesting) data
also doesn't seem to affect this. Changing rebalanceDelay and commenting
out affinityMapper also made no difference.
From what I'm seeing, affinity function distributes partitions evenly (6
nodes, ~341 partitions each = 2048, i.e. 1024 partitions and a backup), but
the coordinator node just never releases 1024-341=683 partitions, being an
owner of every partition in a grid.
Please, help me understand what might cause such behavior. I included logs
and properties, which seemed relevant to the issue, but I'll provide more
if needed.
- regards, Sergii
--
The information in this message may be confidential. It is intended solely
for
the addressee(s). If you are not the intended recipient, any disclosure,
copying or distribution of the message, or any action or omission taken by
you
in reliance on it, is prohibited and may be unlawful. Please immediately
contact the sender if you have received this message in error.
Re: Rebalance is skipped
Posted by Sergii Tyshlek <st...@llnw.com>.
Sorry for bothering. Turns out, it was our misconfiguration.
It's unlikely that someone will have similar issue, but still, here's what
happened:
In order to fix another issue (circular references and lazy-loading in our
components), some time ago we changed cache mode of atomic cache to LOCAL.
At that point it did fixed our problem, but caused another.
The moment I changed cache mode back to defaults, both issues were fixed
(maybe resolution of the second issue has something to do with Ignite
upgrade to 1.7, not sure).
Looks like 'local' mode of atomic cache was blocking rebalancing of all the
caches, which is still strange (at least, not obvious).
So, probably it would be a good idea to point out somewhere, that system
caches' mode should not be changed.
On Tue, Aug 16, 2016 at 12:54 PM, Sergii Tyshlek <st...@llnw.com> wrote:
> Hello there!
>
> Some time ago we started moving from old GridGain to current Apache Ignite
> (1.6, now 1.7).
>
> Here are some cache config properties we use:
>
> --------------------------------------------
> cacheMode=PARTITIONED
> atomicityMode=ATOMIC
> atomicWriteOrderMode=PRIMARY
> writeSynchronizationMode=PRIMARY_SYNC
>
> rebalanceMode=ASYNC
> rebalanceBatchesPrefetchCount=2
> rebalanceDelay=30000
> rebalanceTimeout=10000
>
> backups=1
> affinity=FairAffinityFunction
> affinity.partitions=1024
> --------------------------------------------
>
> At first, everything looked OK, but then I noticed that our data is not
> distributed evenly between nodes (despite the fact we use
> FairAffinityFunction, coordinator node hoards most of the cache entries).
> Later I discovered (through wrapping my custom class around
> FairAffinityFunction) that it works as expected, but the rebalancing is not.
>
> Short after starting 6 nodes (after the last one joins the topology), such
> debug logs appear:
>
> ---------------------------------------------------------
> 2016-08-15 07:36:43,070 DEBUG [exchange-worker-#318%EQGrid%]
> [preloader.GridDhtPreloader] - <p_queryResults> Skipping partition
> assignment (state is not MOVING): GridDhtLocalPartition [id=0,
> map=org.apache.ignite.internal.processors.cache.GridCacheCon
> currentMapImpl@5c35248d, rmvQueue=GridCircularBuffer [sizeMask=255,
> idxGen=0], cntr=0, state=OWNING, reservations=0, empty=true,
> createTime=08/15/2016 07:36:33]
> ...
> // repeats total of 1024 times, where id=0..1023, one for every partition
> // then it's followed by
> 2016-08-15 07:36:43,177 DEBUG [exchange-worker-#318%EQGrid%]
> [preloader.GridDhtPartitionDemander] - <p_queryResults> Adding partition
> assignments: GridDhtPreloaderAssignments [topVer=AffinityTopologyVersion
> [topVer=6, minorTopVer=0], cancelled=false, exchId=GridDhtPartitionExchangeId
> [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0],
> nodeId=383991fb, evt=NODE_JOINED], super={}]
> 2016-08-15 07:36:43,177 DEBUG [exchange-worker-#318%EQGrid%]
> [preloader.GridDhtPartitionDemander] - <p_queryResults> Rebalancing is
> not required [cache=p_queryResults, topology=AffinityTopologyVersion
> [topVer=6, minorTopVer=0]]
> 2016-08-15 07:36:43,178 DEBUG [exchange-worker-#318%EQGrid%]
> [preloader.GridDhtPartitionDemander] - <p_queryResults> Completed
> rebalance future: RebalanceFuture [sndStoppedEvnt=false,
> topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], updateSeq=6]
> 2016-08-15 07:36:43,179 INFO [exchange-worker-#318%EQGrid%]
> [cache.GridCachePartitionExchangeManager] - Skipping rebalancing (nothing
> scheduled) [top=AffinityTopologyVersion [topVer=6, minorTopVer=0],
> evt=NODE_JOINED, node=383991fb-5453-4893-9040-1baa1291881a]
> ---------------------------------------------------------
>
> So I started digging. Using GridDhtPartitionTopology, I got partitions
> map, which (aggregated) looked like this:
> ---------------------------------------------------------
> Node: 38ae4165-474d-4ed4-a292-cca78b8df5c3, partitions: {MOVING=340}
> Node: 8ac8d327-dc59-473f-a3e1-c5861f63f0e6, partitions: {MOVING=341}
> Node: c7047158-9e7b-494f-bceb-3a5774853a6c, partitions: {MOVING=342}
> Node: c9cc1a1f-f037-43c8-8855-0f1ccb8f0ec5, partitions: {MOVING=342}
> Node: dce874ff-cc1e-41c8-9e82-abfb3dfa535e, partitions: {OWNING=1024}
> Node: de783f6d-dc48-46b8-a387-91dd3d181150, partitions: {MOVING=342}
> ---------------------------------------------------------
>
> Important point is that such distribution never changes, neither right
> after grid start, nor after few hours. Ingesting (or not ingesting) data
> also doesn't seem to affect this. Changing rebalanceDelay and commenting
> out affinityMapper also made no difference.
> From what I'm seeing, affinity function distributes partitions evenly (6
> nodes, ~341 partitions each = 2048, i.e. 1024 partitions and a backup), but
> the coordinator node just never releases 1024-341=683 partitions, being an
> owner of every partition in a grid.
>
> Please, help me understand what might cause such behavior. I included logs
> and properties, which seemed relevant to the issue, but I'll provide more
> if needed.
>
> - regards, Sergii
>
--
[image: Limelight Networks] <http://www.limelight.com>
Sergii Tyshlek - *Engineer*
+380669184839
www.limelight.com Delivering Faster Better
Join the conversation
at Limelight Connect <https://connect.limelight.com/>
[image: Facebook] <https://connect.limelight.com/>
[image: Facebook] <https://www.facebook.com/LimelightNetworks> [image:
LinkedIn] <http://www.linkedin.com/company/limelight-networks> [image:
Twitter] <https://twitter.com/llnw>
--
The information in this message may be confidential. It is intended solely
for
the addressee(s). If you are not the intended recipient, any disclosure,
copying or distribution of the message, or any action or omission taken by
you
in reliance on it, is prohibited and may be unlawful. Please immediately
contact the sender if you have received this message in error.
Re: Rebalance is skipped
Posted by Eduard Shangareev <ed...@gmail.com>.
Hi, Sergii.
I can't reproduce this issue. I have written test.
https://github.com/EdShangGG/ignite/commit/8a9462c3a55c6c0317fb8bdc9ef26e12fdbcfb9a
It uses the same configuration as you mentioned.
Could you take a look? Maybe, I am missing something.
On Tue, Aug 16, 2016 at 12:54 PM, Sergii Tyshlek <st...@llnw.com> wrote:
> Hello there!
>
> Some time ago we started moving from old GridGain to current Apache Ignite
> (1.6, now 1.7).
>
> Here are some cache config properties we use:
>
> --------------------------------------------
> cacheMode=PARTITIONED
> atomicityMode=ATOMIC
> atomicWriteOrderMode=PRIMARY
> writeSynchronizationMode=PRIMARY_SYNC
>
> rebalanceMode=ASYNC
> rebalanceBatchesPrefetchCount=2
> rebalanceDelay=30000
> rebalanceTimeout=10000
>
> backups=1
> affinity=FairAffinityFunction
> affinity.partitions=1024
> --------------------------------------------
>
> At first, everything looked OK, but then I noticed that our data is not
> distributed evenly between nodes (despite the fact we use
> FairAffinityFunction, coordinator node hoards most of the cache entries).
> Later I discovered (through wrapping my custom class around
> FairAffinityFunction) that it works as expected, but the rebalancing is not.
>
> Short after starting 6 nodes (after the last one joins the topology), such
> debug logs appear:
>
> ---------------------------------------------------------
> 2016-08-15 07:36:43,070 DEBUG [exchange-worker-#318%EQGrid%]
> [preloader.GridDhtPreloader] - <p_queryResults> Skipping partition
> assignment (state is not MOVING): GridDhtLocalPartition [id=0,
> map=org.apache.ignite.internal.processors.cache.GridCacheCon
> currentMapImpl@5c35248d, rmvQueue=GridCircularBuffer [sizeMask=255,
> idxGen=0], cntr=0, state=OWNING, reservations=0, empty=true,
> createTime=08/15/2016 07:36:33]
> ...
> // repeats total of 1024 times, where id=0..1023, one for every partition
> // then it's followed by
> 2016-08-15 07:36:43,177 DEBUG [exchange-worker-#318%EQGrid%]
> [preloader.GridDhtPartitionDemander] - <p_queryResults> Adding partition
> assignments: GridDhtPreloaderAssignments [topVer=AffinityTopologyVersion
> [topVer=6, minorTopVer=0], cancelled=false, exchId=GridDhtPartitionExchangeId
> [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0],
> nodeId=383991fb, evt=NODE_JOINED], super={}]
> 2016-08-15 07:36:43,177 DEBUG [exchange-worker-#318%EQGrid%]
> [preloader.GridDhtPartitionDemander] - <p_queryResults> Rebalancing is
> not required [cache=p_queryResults, topology=AffinityTopologyVersion
> [topVer=6, minorTopVer=0]]
> 2016-08-15 07:36:43,178 DEBUG [exchange-worker-#318%EQGrid%]
> [preloader.GridDhtPartitionDemander] - <p_queryResults> Completed
> rebalance future: RebalanceFuture [sndStoppedEvnt=false,
> topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], updateSeq=6]
> 2016-08-15 07:36:43,179 INFO [exchange-worker-#318%EQGrid%]
> [cache.GridCachePartitionExchangeManager] - Skipping rebalancing (nothing
> scheduled) [top=AffinityTopologyVersion [topVer=6, minorTopVer=0],
> evt=NODE_JOINED, node=383991fb-5453-4893-9040-1baa1291881a]
> ---------------------------------------------------------
>
> So I started digging. Using GridDhtPartitionTopology, I got partitions
> map, which (aggregated) looked like this:
> ---------------------------------------------------------
> Node: 38ae4165-474d-4ed4-a292-cca78b8df5c3, partitions: {MOVING=340}
> Node: 8ac8d327-dc59-473f-a3e1-c5861f63f0e6, partitions: {MOVING=341}
> Node: c7047158-9e7b-494f-bceb-3a5774853a6c, partitions: {MOVING=342}
> Node: c9cc1a1f-f037-43c8-8855-0f1ccb8f0ec5, partitions: {MOVING=342}
> Node: dce874ff-cc1e-41c8-9e82-abfb3dfa535e, partitions: {OWNING=1024}
> Node: de783f6d-dc48-46b8-a387-91dd3d181150, partitions: {MOVING=342}
> ---------------------------------------------------------
>
> Important point is that such distribution never changes, neither right
> after grid start, nor after few hours. Ingesting (or not ingesting) data
> also doesn't seem to affect this. Changing rebalanceDelay and commenting
> out affinityMapper also made no difference.
> From what I'm seeing, affinity function distributes partitions evenly (6
> nodes, ~341 partitions each = 2048, i.e. 1024 partitions and a backup), but
> the coordinator node just never releases 1024-341=683 partitions, being an
> owner of every partition in a grid.
>
> Please, help me understand what might cause such behavior. I included logs
> and properties, which seemed relevant to the issue, but I'll provide more
> if needed.
>
> - regards, Sergii
>
> The information in this message may be confidential. It is intended
> solely for
> the addressee(s). If you are not the intended recipient, any disclosure,
> copying or distribution of the message, or any action or omission taken by
> you
> in reliance on it, is prohibited and may be unlawful. Please immediately
> contact the sender if you have received this message in error.
>
>