Kafka SinkConnector for K-V store and Keys

[Nilkanth Patel <ni...@gmail.com>]

Hello,

I am writing a kafka sink connector for my product that is distributed
table (Underneath a distribute K-V store where cluster of nodes are holding
a different partitions/buckets of a table. This is based on Hash
partitioned on Keys).

When i write a SinkTask, I get the SinkRecord that contains both key and
value.

Can some one suggest/outline the general guidelines for keys to be used
with K-V store from the SinkRecord.

What should be the key for external K-V store to be used to store a records
from kafka topics to external K-V store.

Thanks,
Nilkanth.

Re: Kafka SinkConnector for K-V store and Keys

[Gwen Shapira <gw...@confluent.io>]

This may be a bit use-case dependent, but I think simply using the key
from the Kafka record as the KV key is a good start.
Another option is to use topic-partition-offset as the key. This has
the benefit of removing duplicates, but it also means that keys are no
longer meaningful for applications querying the KV store.

The nice thing is that now that we are adding transformations of
Connect, you can build one solution and users who don't like it can
use transformations to modify the key.

Gwen

On Tue, Jan 10, 2017 at 8:02 PM, Nilkanth Patel
<ni...@gmail.com> wrote:
> Hello,
>
> I am writing a kafka sink connector for my product that is distributed
> table (Underneath a distribute K-V store where cluster of nodes are holding
> a different partitions/buckets of a table. This is based on Hash
> partitioned on Keys).
>
> When i write a SinkTask, I get the SinkRecord that contains both key and
> value.
>
> Can some one suggest/outline the general guidelines for keys to be used
> with K-V store from the SinkRecord.
>
> What should be the key for external K-V store to be used to store a records
> from kafka topics to external K-V store.
>
> Thanks,
> Nilkanth.



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog