Mantain only last message by key

[Jesus Cabrera Reveles <je...@encontrack.com>]

Hello, 
We are a company of IoT and we are trying to implement kafka but we have some questions. 


We need a topic with a certain number of partitions, each message has its own key, relative to id device. And we need the topic hold only the last message, we don't need historic of messages with the same key. 
It is possible configure the topic with this characteristic? 


Thanks 


-- 


Jesús Cabrera Reveles 
Programador Web 
jesus.cabrera@encontrack.com 
www.encontrack.com 
Querétaro 






Aviso de Confidencialidad: 


El presente mensaje y material adjunto es confidencial, la información aquí contenida está legalmente protegida y su divulgación indebida será sancionada. Esta dirigido únicamente al destinatario. Si usted no es el destinatario, no deberá copiarlo, revelarlo o distribuirlo. Cualquier acción realizada en este sentido, será ilegal. Si por error recibe el presente mensaje, por favor notifique al remitente. 

Re: Mantain only last message by key

[Kevin A <k4...@gmail.com>]

+1. Sounds like what you want is a compacted topic. Check out this section
of the docs: https://kafka.apache.org/documentation#compaction

Keep in mind that compaction isn't free. Like a garbage collector, it comes
with some overhead and some knobs for tuning.

On Wed, Oct 19, 2016 at 2:46 PM, Radoslaw Gruchalski <ra...@gruchalski.com>
wrote:

> You can try cleanup.policy=compact.
> But be careful with a large number of keys.
>
> –
> Best regards,
> Radek Gruchalski
> radek@gruchalski.com
>
>
> On October 19, 2016 at 11:44:39 PM, Jesus Cabrera Reveles (
> jesus.cabrera@encontrack.com) wrote:
>
>
> Hello,
> We are a company of IoT and we are trying to implement kafka but we have
> some questions.
>
>
> We need a topic with a certain number of partitions, each message has its
> own key, relative to id device. And we need the topic hold only the last
> message, we don't need historic of messages with the same key.
> It is possible configure the topic with this characteristic?
>
>
> Thanks
>
>
> --
>
>
> Jesús Cabrera Reveles
> Programador Web
> jesus.cabrera@encontrack.com
> www.encontrack.com
> Querétaro
>
>
>
>
>
>
> Aviso de Confidencialidad:
>
>
> El presente mensaje y material adjunto es confidencial, la información aquí
> contenida está legalmente protegida y su divulgación indebida será
> sancionada. Esta dirigido únicamente al destinatario. Si usted no es el
> destinatario, no deberá copiarlo, revelarlo o distribuirlo. Cualquier
> acción realizada en este sentido, será ilegal. Si por error recibe el
> presente mensaje, por favor notifique al remitente.
>

Re: Mantain only last message by key

[Radoslaw Gruchalski <ra...@gruchalski.com>]

You can try cleanup.policy=compact.
But be careful with a large number of keys.

–
Best regards,
Radek Gruchalski
radek@gruchalski.com


On October 19, 2016 at 11:44:39 PM, Jesus Cabrera Reveles (
jesus.cabrera@encontrack.com) wrote:


Hello,
We are a company of IoT and we are trying to implement kafka but we have
some questions.


We need a topic with a certain number of partitions, each message has its
own key, relative to id device. And we need the topic hold only the last
message, we don't need historic of messages with the same key.
It is possible configure the topic with this characteristic?


Thanks


-- 


Jesús Cabrera Reveles
Programador Web
jesus.cabrera@encontrack.com
www.encontrack.com
Querétaro






Aviso de Confidencialidad:


El presente mensaje y material adjunto es confidencial, la información aquí
contenida está legalmente protegida y su divulgación indebida será
sancionada. Esta dirigido únicamente al destinatario. Si usted no es el
destinatario, no deberá copiarlo, revelarlo o distribuirlo. Cualquier
acción realizada en este sentido, será ilegal. Si por error recibe el
presente mensaje, por favor notifique al remitente.