You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "aaron@tophold.com" <aa...@tophold.com> on 2017/10/18 07:45:23 UTC
Where can we get the partition assignment of a Cache after the cluster changed?
hi All,
if there any events trigger when a cache's partition assignment updates.
Usually this happen before or after the EVTS_DISCOVERY? we may get the partition assignment after catch this events.
we know there is a #primaryPartitions API, but be aware each cache node's primary partitions is important to our use case.
Thanks for your time!
Regards
Aaron
aaron@tophold.com
Re: Re: Where can we get the partition assignment of a Cache after
the cluster changed?
Posted by "slava.koptilin" <sl...@gmail.com>.
Hi Aaron,
> If I bring up the node X, every events as expected, but Now If I start
> Node Y,
> I still got a lot CacheA and CacheB 's rebalance events.
> But Node Y's start or stop should not affect CacheA & CacheB,
> their partition never change or rebalance, log after Y started:
Yes, I see your point.
When a new node joins the cluster the process of 'partition map exchange' is
triggered
(potentially, some partitions will be moved to a new node) and this the
reason why you get CACHE_REBALACE_STARTED/STOPPED events.
In your particular case, these messages are redundant. Perhaps, it should be
handled on Ignite side...
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re: Where can we get the partition assignment of a Cache after the cluster changed?
Posted by "aaron@tophold.com" <aa...@tophold.com>.
hi Slava,
Thanks for detail, I adjust a bit, CacheA and CacheB host on first node with NodeType as X
<property name="userAttributes">
<map><entry key="NodeType" value="X"/></map>
</property>
While CacheC host on another node which NodeType as Y, when create cache configuration I let CacheA&B only on Node X, and CacheC on node Y;
CacheConfiguration ccfg = new CacheConfiguration(name);
ccfg.setCacheMode(CacheMode.PARTITIONED);
ccfg.setEagerTtl(true);
ccfg.setName(name);
ccfg.setNodeFilter(new AttributeNodeFilter("NodeType", nodeType));
If I bring up the node X, every events as expected, but Now If I start Node Y, I still got a lot CacheA and CacheB 's rebalance events.
But Node Y's start or stop should not affect CacheA & CacheB, their partition never change or rebalance, log after Y started:
[11:05:46] Topology snapshot [ver=2, servers=2, clients=0, CPUs=8, heap=18.0GB]
Received event [evt=CACHE_REBALANCE_STARTED,evt=CacheRebalancingEvent [cacheName=CacheA, part=-1, discoNode=TcpDiscoveryNode [id=55063d61-4168-4727-a5b9-1e1622fb57aa, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.10.36, 192.168.56.1, 2001:0:d388:7101:1c4c:155:3f57:f5db], sockAddrs=[Aaron/192.168.10.36:47501, /2001:0:d388:7101:1c4c:155:3f57:f5db:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501, /192.168.56.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1508900746905, loc=false, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], discoEvtType=10, discoTs=1508900746975, discoEvtName=NODE_JOINED, nodeId8=16c5f906, msg=Cache rebalancing event., type=CACHE_REBALANCE_STARTED, tstamp=1508900747387]
Received event [evt=CACHE_REBALANCE_STOPPED,evt=CacheRebalancingEvent [cacheName=CacheA, part=-1, discoNode=TcpDiscoveryNode [id=55063d61-4168-4727-a5b9-1e1622fb57aa, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.10.36, 192.168.56.1, 2001:0:d388:7101:1c4c:155:3f57:f5db], sockAddrs=[Aaron/192.168.10.36:47501, /2001:0:d388:7101:1c4c:155:3f57:f5db:47501, /0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501, /192.168.56.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1508900746905, loc=false, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], discoEvtType=10, discoTs=1508900746975, discoEvtName=NODE_JOINED, nodeId8=16c5f906, msg=Cache rebalancing event., type=CACHE_REBALANCE_STOPPED, tstamp=1508900747387]
Regards
Aaron
aaron@tophold.com
From: slava.koptilin
Date: 2017-10-24 20:29
To: user
Subject: Re: Re: Where can we get the partition assignment of a Cache after the cluster changed?
Hi Aaron,
I just started a single Ignite node as follows:
Ignite ignite =
Ignition.start("examples/config/example-ignite.xml");
IgnitePredicate<Event> rebalanceEventLsnr = evt -> {
System.out.println("Received event [evt=" + evt.name() + ",
evt=" + evt.toString());
return true; // Continue listening.
};
ignite.events().localListen(rebalanceEventLsnr,
EventType.EVT_CACHE_REBALANCE_STOPPED,
EventType.EVT_CACHE_REBALANCE_STARTED);
Thread.sleep(5000);
IgniteCache cache =
ignite.getOrCreateCache(createCacheConfiguration("CacheA"));
IgniteCache cache2 =
ignite.getOrCreateCache(createCacheConfiguration("CacheB"));
and got the following output:
[15:22:39,460][INFO ][exchange-worker-#31%null%][GridCacheProcessor] Started
cache [name=CacheA, mode=PARTITIONED]
Received event [evt=CACHE_REBALANCE_STARTED, evt=CacheRebalancingEvent
[cacheName=CacheA, part=-1, discoNode=TcpDiscoveryNode [...]]
Received event [evt=CACHE_REBALANCE_STOPPED, evt=CacheRebalancingEvent
[cacheName=CacheA, part=-1, discoNode=TcpDiscoveryNode [...]]
[15:22:39,478][INFO
][exchange-worker-#31%null%][GridCachePartitionExchangeManager] Skipping
rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=1,
minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT,
node=1ee1a908-2424-46b0-911b-0a2c70ad68e2]
[15:22:39,480][INFO ][exchange-worker-#31%null%][GridCacheProcessor] Started
cache [name=CacheB, mode=PARTITIONED]
Received event [evt=CACHE_REBALANCE_STARTED, evt=CacheRebalancingEvent
[cacheName=CacheA, part=-1, discoNode=TcpDiscoveryNode [...]]
Received event [evt=CACHE_REBALANCE_STOPPED, evt=CacheRebalancingEvent
[cacheName=CacheA, part=-1, discoNode=TcpDiscoveryNode [...]]
Received event [evt=CACHE_REBALANCE_STARTED, evt=CacheRebalancingEvent
[cacheName=CacheB, part=-1, discoNode=TcpDiscoveryNode [...]]
Received event [evt=CACHE_REBALANCE_STOPPED, evt=CacheRebalancingEvent
[cacheName=CacheB, part=-1, discoNode=TcpDiscoveryNode [...]]
[15:22:39,485][INFO
][exchange-worker-#31%null%][GridCachePartitionExchangeManager] Skipping
rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=1,
minorTopVer=2], evt=DISCOVERY_CUSTOM_EVT,
node=1ee1a908-2424-46b0-911b-0a2c70ad68e2]
So, I see messages for both caches.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re: Where can we get the partition assignment of a Cache after
the cluster changed?
Posted by "slava.koptilin" <sl...@gmail.com>.
Hi Aaron,
I just started a single Ignite node as follows:
Ignite ignite =
Ignition.start("examples/config/example-ignite.xml");
IgnitePredicate<Event> rebalanceEventLsnr = evt -> {
System.out.println("Received event [evt=" + evt.name() + ",
evt=" + evt.toString());
return true; // Continue listening.
};
ignite.events().localListen(rebalanceEventLsnr,
EventType.EVT_CACHE_REBALANCE_STOPPED,
EventType.EVT_CACHE_REBALANCE_STARTED);
Thread.sleep(5000);
IgniteCache cache =
ignite.getOrCreateCache(createCacheConfiguration("CacheA"));
IgniteCache cache2 =
ignite.getOrCreateCache(createCacheConfiguration("CacheB"));
and got the following output:
[15:22:39,460][INFO ][exchange-worker-#31%null%][GridCacheProcessor] Started
cache [name=CacheA, mode=PARTITIONED]
Received event [evt=CACHE_REBALANCE_STARTED, evt=CacheRebalancingEvent
[cacheName=CacheA, part=-1, discoNode=TcpDiscoveryNode [...]]
Received event [evt=CACHE_REBALANCE_STOPPED, evt=CacheRebalancingEvent
[cacheName=CacheA, part=-1, discoNode=TcpDiscoveryNode [...]]
[15:22:39,478][INFO
][exchange-worker-#31%null%][GridCachePartitionExchangeManager] Skipping
rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=1,
minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT,
node=1ee1a908-2424-46b0-911b-0a2c70ad68e2]
[15:22:39,480][INFO ][exchange-worker-#31%null%][GridCacheProcessor] Started
cache [name=CacheB, mode=PARTITIONED]
Received event [evt=CACHE_REBALANCE_STARTED, evt=CacheRebalancingEvent
[cacheName=CacheA, part=-1, discoNode=TcpDiscoveryNode [...]]
Received event [evt=CACHE_REBALANCE_STOPPED, evt=CacheRebalancingEvent
[cacheName=CacheA, part=-1, discoNode=TcpDiscoveryNode [...]]
Received event [evt=CACHE_REBALANCE_STARTED, evt=CacheRebalancingEvent
[cacheName=CacheB, part=-1, discoNode=TcpDiscoveryNode [...]]
Received event [evt=CACHE_REBALANCE_STOPPED, evt=CacheRebalancingEvent
[cacheName=CacheB, part=-1, discoNode=TcpDiscoveryNode [...]]
[15:22:39,485][INFO
][exchange-worker-#31%null%][GridCachePartitionExchangeManager] Skipping
rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=1,
minorTopVer=2], evt=DISCOVERY_CUSTOM_EVT,
node=1ee1a908-2424-46b0-911b-0a2c70ad68e2]
So, I see messages for both caches.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re: Where can we get the partition assignment of a Cache after the cluster changed?
Posted by "aaron@tophold.com" <aa...@tophold.com>.
Thanks Slava, yes after add those configuration, eventually I can receive those notification.
But seem all the cache's rebalanced event will be published to me with cache name as the cache I monitored.
Like I only care the CacheA 's events:
final CacheRebalancingEvent cre = (CacheRebalancingEvent) event;
if ("CacheA".equals(cre.cacheName())) {
}
But now even the CacheB join the cluster, I still get one event with 'CacheA' as #cacheName.
If I compare local node's primary partition for CacheA, it's not changed actually.
Regards
Aaron
aaron@tophold.com
From: slava.koptilin
Date: 2017-10-24 18:39
To: user
Subject: Re: Re: Where can we get the partition assignment of a Cache after the cluster changed?
Hi Aaron,
Please check that CACHE_REBALANCE events are registered in
IgniteConfiguration via xml or java code (by default, all these events are
disabled).
<property name="includeEventTypes">
<list>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_STARTED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_STOPPED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_PART_LOADED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_PART_UNLOADED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_OBJECT_LOADED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_OBJECT_UNLOADED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST"/>
</list>
</property>
Could you try the following code?
IgnitePredicate<Event> rebalanceEventLsnr = evt -> {
System.out.println("Received event [evt=" + evt.name() + ",
evt=" + evt.toString());
if (evt instanceof CacheRebalancingEvent) {
CacheRebalancingEvent rebalancingEvt =
(CacheRebalancingEvent) evt;
if (rebalancingEvt.cacheName().equals(IG_CACHE_NAME)) {
IgniteCache c = ignite.cache(IG_CACHE_NAME);
ClusterNode localNode = ignite.cluster().localNode();
int[] backups =
ignite.affinity(c.getName()).backupPartitions(localNode);
int[] primaries =
ignite.affinity(c.getName()).primaryPartitions(localNode);
System.out.println("Local node : " + localNode.id());
System.out.println("\t primary: " +
Arrays.toString(primaries));
System.out.println("\t backups: " +
Arrays.toString(backups));
System.out.println("-----------------------------------------");
}
}
return true; // Continue listening.
};
ignite.events().localListen(rebalanceEventLsnr,
EventType.EVT_CACHE_REBALANCE_STOPPED);
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re: Where can we get the partition assignment of a Cache after
the cluster changed?
Posted by "slava.koptilin" <sl...@gmail.com>.
Hi Aaron,
Please check that CACHE_REBALANCE events are registered in
IgniteConfiguration via xml or java code (by default, all these events are
disabled).
<property name="includeEventTypes">
<list>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_STARTED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_STOPPED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_PART_LOADED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_PART_UNLOADED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_OBJECT_LOADED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_OBJECT_UNLOADED"/>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST"/>
</list>
</property>
Could you try the following code?
IgnitePredicate<Event> rebalanceEventLsnr = evt -> {
System.out.println("Received event [evt=" + evt.name() + ",
evt=" + evt.toString());
if (evt instanceof CacheRebalancingEvent) {
CacheRebalancingEvent rebalancingEvt =
(CacheRebalancingEvent) evt;
if (rebalancingEvt.cacheName().equals(IG_CACHE_NAME)) {
IgniteCache c = ignite.cache(IG_CACHE_NAME);
ClusterNode localNode = ignite.cluster().localNode();
int[] backups =
ignite.affinity(c.getName()).backupPartitions(localNode);
int[] primaries =
ignite.affinity(c.getName()).primaryPartitions(localNode);
System.out.println("Local node : " + localNode.id());
System.out.println("\t primary: " +
Arrays.toString(primaries));
System.out.println("\t backups: " +
Arrays.toString(backups));
System.out.println("-----------------------------------------");
}
}
return true; // Continue listening.
};
ignite.events().localListen(rebalanceEventLsnr,
EventType.EVT_CACHE_REBALANCE_STOPPED);
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re: Where can we get the partition assignment of a Cache after the cluster changed?
Posted by "aaron@tophold.com" <aa...@tophold.com>.
Plus I monitor those Events too:
EVT_NODE_JOINED,
EVT_NODE_LEFT,
EVT_NODE_FAILED,
EVT_NODE_SEGMENTEDSeem the primary partitions support by local node is still wrong. Every time I bring up ignite instance print:for (int part : ignite.affinity(IG_CACHE_NAME).primaryPartitions(ignite.cluster().localNode())) {
logger.error("Init Primary: {}", part);
}And after node events trigger, I print it again:for (int part : ignite.affinity(IG_CACHE_NAME).primaryPartitions(ignite.cluster().localNode())) {
logger.error("Current Primary: {}", part);
}and have 3 nodes with partitions number 16, bring up them and kill and re-bring up , the primary partitions seem never match, between no EVT_CACHE_REBALANCE* events received. 1, Bring them up one by one A:
Init Primary: 0
Init Primary: 1
Init Primary: 2
Init Primary: 3
Init Primary: 4
Init Primary: 5
Init Primary: 6
Init Primary: 7
Init Primary: 8
Init Primary: 9
Init Primary: 10
Init Primary: 11
Init Primary: 12
Init Primary: 13
Init Primary: 14
Init Primary: 15
B Join
Noting change
C: Join A:
Current Primary: 1
Current Primary: 2
Current Primary: 3
Current Primary: 4
Current Primary: 6
Current Primary: 8
Current Primary: 9
Current Primary: 10
Current Primary: 11
Current Primary: 12
Current Primary: 13
Current Primary: 14
B:
Current Primary: 0
Current Primary: 5
Current Primary: 7
Current Primary: 15
C:
Nothing change
------------------------
2. KILL A:
B:
Current Primary: 0
Current Primary: 5
Current Primary: 7
C:
Current Primary: 10
Current Primary: 14
Current Primary: 15
-----------------------
3. KILL B:
C:
Current Primary: 2
Current Primary: 10
Current Primary: 12
Current Primary: 13
Current Primary: 14
Current Primary: 15
----------------------
4. Bring up B:
B:
Init Primary: 0
Init Primary: 1
Init Primary: 2
Init Primary: 3
Init Primary: 4
Init Primary: 5
Init Primary: 6
Init Primary: 8
Init Primary: 9
Init Primary: 11
Init Primary: 12
Init Primary: 13
C:
Current Primary: 0
Current Primary: 1
Current Primary: 2
Current Primary: 3
Current Primary: 4
Current Primary: 5
Current Primary: 6
Current Primary: 7
Current Primary: 8
Current Primary: 9
Current Primary: 10
Current Primary: 11
Current Primary: 12
Current Primary: 13
Current Primary: 14
Current Primary: 15
----------------------
5. Bring up A:
A:Nothing print
B:
Current Primary: 0
Current Primary: 1
Current Primary: 2
Current Primary: 3
Current Primary: 4
Current Primary: 5
Current Primary: 6
Current Primary: 8
Current Primary: 9
Current Primary: 11
Current Primary: 12
Current Primary: 13
C:
Current Primary: 7
Current Primary: 10
Current Primary: 14
Current Primary: 15
Regards
Aaron
aaron@tophold.com
From: aaron@tophold.com
Date: 2017-10-19 10:48
To: user
Subject: Re: Re: Where can we get the partition assignment of a Cache after the cluster changed?
hi Slava,
I monitor all those events:
EVT_CACHE_REBALANCE_PART_LOADED,
EVT_CACHE_REBALANCE_PART_UNLOADED,
EVT_CACHE_REBALANCE_STARTED,
EVT_CACHE_REBALANCE_STOPPED
And found when I start multiple nodes, neither of those events be triggered. my cache work as PARTITIONED.
But if I monitor the EVT_NODE_JOINED, I can find the node joined, so now the first node still carry all the primary partitions.
This is not what we expected, we want those two nodes to do load balance, so when exactly the REBALANCE started?
Regards
Aaron
aaron@tophold.com
From: aaron@tophold.com
Date: 2017-10-18 19:23
To: user
Subject: Re: Re: Where can we get the partition assignment of a Cache after the cluster changed?
Got thanks Slava!
aaron@tophold.com
From: slava.koptilin
Date: 2017-10-18 17:52
To: user
Subject: Re: Where can we get the partition assignment of a Cache after the cluster changed?
Hi Aaron,
Please take a look at the following event types:
- EventType#EVT_CACHE_REBALANCE_PART_LOADED [1]
- EventType#EVT_CACHE_REBALANCE_PART_UNLOADED [2]
[1]
https://ignite.apache.org/releases/2.2.0/javadoc/org/apache/ignite/events/EventType.html#EVT_CACHE_REBALANCE_PART_LOADED
[2]
https://ignite.apache.org/releases/2.2.0/javadoc/org/apache/ignite/events/EventType.html#EVT_CACHE_REBALANCE_PART_UNLOADED
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re: Where can we get the partition assignment of a Cache after the cluster changed?
Posted by "aaron@tophold.com" <aa...@tophold.com>.
hi Slava,
I monitor all those events:
EVT_CACHE_REBALANCE_PART_LOADED,
EVT_CACHE_REBALANCE_PART_UNLOADED,
EVT_CACHE_REBALANCE_STARTED,
EVT_CACHE_REBALANCE_STOPPED
And found when I start multiple nodes, neither of those events be triggered. my cache work as PARTITIONED.
But if I monitor the EVT_NODE_JOINED, I can find the node joined, so now the first node still carry all the primary partitions.
This is not what we expected, we want those two nodes to do load balance, so when exactly the REBALANCE started?
Regards
Aaron
aaron@tophold.com
From: aaron@tophold.com
Date: 2017-10-18 19:23
To: user
Subject: Re: Re: Where can we get the partition assignment of a Cache after the cluster changed?
Got thanks Slava!
aaron@tophold.com
From: slava.koptilin
Date: 2017-10-18 17:52
To: user
Subject: Re: Where can we get the partition assignment of a Cache after the cluster changed?
Hi Aaron,
Please take a look at the following event types:
- EventType#EVT_CACHE_REBALANCE_PART_LOADED [1]
- EventType#EVT_CACHE_REBALANCE_PART_UNLOADED [2]
[1]
https://ignite.apache.org/releases/2.2.0/javadoc/org/apache/ignite/events/EventType.html#EVT_CACHE_REBALANCE_PART_LOADED
[2]
https://ignite.apache.org/releases/2.2.0/javadoc/org/apache/ignite/events/EventType.html#EVT_CACHE_REBALANCE_PART_UNLOADED
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Re: Where can we get the partition assignment of a Cache after the cluster changed?
Posted by "aaron@tophold.com" <aa...@tophold.com>.
Got thanks Slava!
aaron@tophold.com
From: slava.koptilin
Date: 2017-10-18 17:52
To: user
Subject: Re: Where can we get the partition assignment of a Cache after the cluster changed?
Hi Aaron,
Please take a look at the following event types:
- EventType#EVT_CACHE_REBALANCE_PART_LOADED [1]
- EventType#EVT_CACHE_REBALANCE_PART_UNLOADED [2]
[1]
https://ignite.apache.org/releases/2.2.0/javadoc/org/apache/ignite/events/EventType.html#EVT_CACHE_REBALANCE_PART_LOADED
[2]
https://ignite.apache.org/releases/2.2.0/javadoc/org/apache/ignite/events/EventType.html#EVT_CACHE_REBALANCE_PART_UNLOADED
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Where can we get the partition assignment of a Cache after the
cluster changed?
Posted by "slava.koptilin" <sl...@gmail.com>.
Hi Aaron,
Please take a look at the following event types:
- EventType#EVT_CACHE_REBALANCE_PART_LOADED [1]
- EventType#EVT_CACHE_REBALANCE_PART_UNLOADED [2]
[1]
https://ignite.apache.org/releases/2.2.0/javadoc/org/apache/ignite/events/EventType.html#EVT_CACHE_REBALANCE_PART_LOADED
[2]
https://ignite.apache.org/releases/2.2.0/javadoc/org/apache/ignite/events/EventType.html#EVT_CACHE_REBALANCE_PART_UNLOADED
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/