Re: Kafka & Ignite AffinityFunction

[Mikhail Pochatkin <mp...@unison.team>]

Hi! Looks like you can't achieve it in Ignite 3 the same way as Ignite 2
(at least currently). Current implementation of affinity in Ignite 3 based
on  ignite-3/RendezvousAffinityFunction.java at main · apache/ignite-3
(github.com)
<https://github.com/apache/ignite-3/blob/main/modules/affinity/src/main/java/org/apache/ignite/internal/affinity/RendezvousAffinityFunction.java>
as
said previously. But unfortunately you cannot customize table affinity
function, as in Ignite 2 cache options. So, it looks like the only thing
that you can specify is `COLOCATED BY` - colocation key. The key can be
composite. Primary key must include colocation key. Was `affinity_key` in
Ignite 2.x. when you create your table.

On Thu, Jan 26, 2023 at 3:08 PM Colin Cassidy via user <
user@ignite.apache.org> wrote:

> Thanks for the response, but I'm afraid that a simple affinity key doesn't
> help in this case. To clarify - I don't just want to ensure that data
> records in Ignite are stored in the same node - I want to ensure that the
> events from Kafka arrive on the right node too.
>
> Kafka uses a partitoning and assignment scheme very similar to the one
> Ignite uses. It also supports event affinity. But the actual paritition &
> node assignment will never match up between Kafka and Ignite.
>
> That means that an event e1 might land on a kafka consumer running on
> Ignite node n1 - but Ignite may chose to store the data on node n2. For a
> high throughput of events, this kills performance and scale.
>
> What I want to do is override the assignment to ensure that the event
> landing on node n1 is also stored on node n1. Meaning that the data access
> will normally be local and therefore fast.
>
> Second prize would be to ensure that at least the partitions are aligned.
> So that the Kafka and Ignite partition number for a given key is the same.
> So if k1 & k2 are mapped to partition 1 and k3, k4 are mapped to partition
> 2 - that will be true both for Kafka and Ignite. Then at least the node
> assignment can be handled on the consumer and doesn't require a change to
> the producer.
>
> In any case, I think all of this could be achieved in Ignite 2 - but the
> question is - can it be achieved also in Ignite 3?
>
> Thanks,
> Colin.
>
> On Thursday, 26 January 2023 at 11:57:06 GMT, Stephen Darlington <
> stephen.darlington@gridgain.com> wrote:
>
>
> You don’t need to override the RendezvousAffinityFunction to do what
> you’re asking. Instead, if you define an affinity key to your table, it
> will guarantee that related records will be kept together. However, make
> sure your groups are not too coarse or you’ll get poor distribution across
> your cluster.
>
> On 26 Jan 2023, at 11:37, Colin Cassidy via user <us...@ignite.apache.org>
> wrote:
>
> I would like to override the Ignite AffinityFunction to delegate to
> Kafka's partitioning algorithm. Possibly also the node assignment - but
> definitely the partitioning
>
> This will allow me to ensure that events are processed local to where the
> data is stored.
>
> I think this is possible in Ignite 2.x but there appear to be significant
> changes in Ignite 3.x. Could I confirm - if I follow this path, will it be
> possible to override the cache partitioning still in 3.x?
>
> Thanks in advance,
> Colin.
>
>
>

Re: Kafka & Ignite AffinityFunction

[Colin Cassidy via user <us...@ignite.apache.org>]

 Thanks for confirming. I understand that AffinityFunction was rather dangerous and open to abuse. On the other hand, the ability to align cache & event partitions would be a massive scale unlock for a huge number of use cases. I can't help but think that this might be a missed opportunity.
Regards,Colin.
    On Wednesday, 1 February 2023 at 11:03:29 GMT, Mikhail Pochatkin <mp...@unison.team> wrote:  
 
 Hi! Looks like you can't achieve it in Ignite 3 the same way as Ignite 2 (at least currently). Current implementation of affinity in Ignite 3 based on ignite-3/RendezvousAffinityFunction.java at main · apache/ignite-3 (github.com) as said previously. But unfortunately you cannot customize table affinity function, as in Ignite 2 cache options. So, it looks like the only thing that you can specify is `COLOCATED BY` - colocation key. The key can be composite. Primary key must include colocation key. Was `affinity_key` in Ignite 2.x. when you create your table.

On Thu, Jan 26, 2023 at 3:08 PM Colin Cassidy via user <us...@ignite.apache.org> wrote:

 Thanks for the response, but I'm afraid that a simple affinity key doesn't help in this case. To clarify - I don't just want to ensure that data records in Ignite are stored in the same node - I want to ensure that the events from Kafka arrive on the right node too.
Kafka uses a partitoning and assignment scheme very similar to the one Ignite uses. It also supports event affinity. But the actual paritition & node assignment will never match up between Kafka and Ignite.
That means that an event e1 might land on a kafka consumer running on Ignite node n1 - but Ignite may chose to store the data on node n2. For a high throughput of events, this kills performance and scale.
What I want to do is override the assignment to ensure that the event landing on node n1 is also stored on node n1. Meaning that the data access will normally be local and therefore fast.
Second prize would be to ensure that at least the partitions are aligned. So that the Kafka and Ignite partition number for a given key is the same. So if k1 & k2 are mapped to partition 1 and k3, k4 are mapped to partition 2 - that will be true both for Kafka and Ignite. Then at least the node assignment can be handled on the consumer and doesn't require a change to the producer.
In any case, I think all of this could be achieved in Ignite 2 - but the question is - can it be achieved also in Ignite 3?
Thanks,Colin. 
    On Thursday, 26 January 2023 at 11:57:06 GMT, Stephen Darlington <st...@gridgain.com> wrote:  
 
 You don’t need to override the RendezvousAffinityFunction to do what you’re asking. Instead, if you define an affinity key to your table, it will guarantee that related records will be kept together. However, make sure your groups are not too coarse or you’ll get poor distribution across your cluster.


On 26 Jan 2023, at 11:37, Colin Cassidy via user <us...@ignite.apache.org> wrote:
I would like to override the Ignite AffinityFunction to delegate to Kafka's partitioning algorithm. Possibly also the node assignment - but definitely the partitioning
This will allow me to ensure that events are processed local to where the data is stored.
I think this is possible in Ignite 2.x but there appear to be significant changes in Ignite 3.x. Could I confirm - if I follow this path, will it be possible to override the cache partitioning still in 3.x?
Thanks in advance,Colin.