You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by rishi007bansod <ri...@gmail.com> on 2017/01/28 18:18:57 UTC
Partition mapping to Ignite node
I have few questions regarding partition mapping,1. Is there any way I can
one to one map kafka partitions to ignite node? So that for example, if I
have 2 partitions P1, P2 in kafka and I have ignite partitioned cache on 2
nodes. I want data from partition P1 to be cached on one node and from
partition P2 to the another node. 2. Also is there any way we can change
hash function(make it user defined) in ignite so that we can cache entries
on nodes in cluster as per our requirement?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Partition-mapping-to-Ignite-node-tp10300.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Partition mapping to Ignite node
Posted by vkulichenko <va...@gmail.com>.
AffinityFunctionContext is the input, it provides information about topology
for which the mapping has to be created. assignPartitions() method then has
to return this mapping. Here is the JavaDoc quote for this method:
Returns: Unmodifiable list indexed by partition number. Each element of
array is a collection in which first node is a primary node and other nodes
are backup nodes.
Everything else depends on what exactly you want to achieve. You can put any
logic you want in this method.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Partition-mapping-to-Ignite-node-tp10300p10491.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Partition mapping to Ignite node
Posted by rishi007bansod <ri...@gmail.com>.
I looked at following link,
https://ignite.apache.org/releases/1.7.0/javadoc/org/apache/ignite/cache/affinity/AffinityFunction.html
<https://ignite.apache.org/releases/1.7.0/javadoc/org/apache/ignite/cache/affinity/AffinityFunction.html>
But i am not getting idea, how can i use AffinityFunctionContext to map
certain partition to certain node(i.e. how can i define my own mapping)? can
you give example for same(As above documentation does not explains this part
in detail)?
Thanks.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Partition-mapping-to-Ignite-node-tp10300p10485.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Partition mapping to Ignite node
Posted by vkulichenko <va...@gmail.com>.
Take a look at AffinityFunction interface and its implementations provided by
Ignite. This should give you good understanding of how it works.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Partition-mapping-to-Ignite-node-tp10300p10347.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Partition mapping to Ignite node
Posted by rishi007bansod <ri...@gmail.com>.
Thanks, Can you give example, that how we can use affinityfunction for this
purpose? Or how can we write hashcode() for key to node mapping?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Partition-mapping-to-Ignite-node-tp10300p10328.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Partition mapping to Ignite node
Posted by colinc <co...@yahoo.co.uk>.
Good answer, thanks. We'll use the backup filter to create a dedicated backup
subcluster.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Partition mapping to Ignite node
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
I think that you should just use AffinityBackupFilter, turn exclNeighbors
off. And please try to make it stateless/in sync between nodes. I can't
imagine what happens otherwise.
Regards,
--
Ilya Kasnacheev
пн, 18 янв. 2021 г. в 21:58, colinc <co...@yahoo.co.uk>:
> We have found that the simplest way to achieve this is by overriding the
> Kafka partitioner and assignor - and delegating to Ignite for partition
> assignment and the affinity function respectively. In this way, Ignite
> controls the data to partition and partition to node assignment and Kafka
> reflects this. The AffinityFunction can be supplied with a topology
> (collection of ClusterNode) that reflects the current Kafka consumers.
> Ignite only uses the persistent node ID in the affinity function.
>
> Implications seem to be:
> * There must be at least as many Kafka partitions as Ignite partitions for
> a
> given topic/cache. Ideally an equal number to keep things simple.
> * There may be some remote operations during a topology change.
> * The Ignite RendezvousAffinityFunction.assignPartition() consumes a
> neighborhoodCache map. This used to ensure that primary and backup nodes do
> not land on the same host when exclNeighbors=true. We haven't tested in
> this
> configuration to determine whether it is deterministic.
>
> The final point is important because the we are not using the internal
> Ignite version of the AffinityFunction but rather a version that has been
> created for the purpose of the Kafka assignor. This works fine if the
> AffinityFunction is stateless but the neighborhoodCache means that it
> becomes stateful. If the state of the various AffinityFunction instances is
> not in sync, then potentially invocations might be inconsistent even for
> the
> same topology?
>
> Does anyone have thoughts on viability of the above in case of
> exclNeighbors=true?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Partition mapping to Ignite node
Posted by colinc <co...@yahoo.co.uk>.
We have found that the simplest way to achieve this is by overriding the
Kafka partitioner and assignor - and delegating to Ignite for partition
assignment and the affinity function respectively. In this way, Ignite
controls the data to partition and partition to node assignment and Kafka
reflects this. The AffinityFunction can be supplied with a topology
(collection of ClusterNode) that reflects the current Kafka consumers.
Ignite only uses the persistent node ID in the affinity function.
Implications seem to be:
* There must be at least as many Kafka partitions as Ignite partitions for a
given topic/cache. Ideally an equal number to keep things simple.
* There may be some remote operations during a topology change.
* The Ignite RendezvousAffinityFunction.assignPartition() consumes a
neighborhoodCache map. This used to ensure that primary and backup nodes do
not land on the same host when exclNeighbors=true. We haven't tested in this
configuration to determine whether it is deterministic.
The final point is important because the we are not using the internal
Ignite version of the AffinityFunction but rather a version that has been
created for the purpose of the Kafka assignor. This works fine if the
AffinityFunction is stateless but the neighborhoodCache means that it
becomes stateful. If the state of the various AffinityFunction instances is
not in sync, then potentially invocations might be inconsistent even for the
same topology?
Does anyone have thoughts on viability of the above in case of
exclNeighbors=true?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Partition mapping to Ignite node
Posted by vkulichenko <va...@gmail.com>.
1. You will need to make sure that partitioning strategy is the same in
Ignite and Kafka. I believe it can be possible, but haven't seen any
examples of this.
2. You can always implement your own AffinityFunction. This is not a trivial
task though, you need to make sure everything works as you expect in case of
topology changes.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Partition-mapping-to-Ignite-node-tp10300p10322.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.