logical delete from a log compacted topic; tombstone message? with kafka-streams

[Nicolae Marasoiu <ni...@ovoenergy.com>]

Hello kafka community,
Is there a way from kafka streams to emit a tombstone? say we have an
events topic with upsert or delete; and an output topic with the current
state only, log compacted; an upsert event will get mapped to current user
topic (it holds the full entity); but a delete would perhaps need to emit a
tombstone message (is there such a thing?) so that the “record” is
effectively / logically deleted from the current state compacted log
What would be the manner to achieve this using kafka-streams?
Thank you,

-- 
Thank you,
Nicolae Marasoiu
Scala Engineer
Orion, OVO Group

Re: logical delete from a log compacted topic; tombstone message? with kafka-streams

[Guozhang Wang <wa...@gmail.com>]

Hello Nicolae,

If your output topic is configured as log compacted, then sending a record
with null-bytes effectively serves as a tombstone. Note that you'd need to
make sure in your sink node's serializer that the serialized bytes are null
when the unserialized object typed value indicates to be a tombstone.


Guozhang


On Thu, Apr 16, 2020 at 10:49 AM Nicolae Marasoiu <
nicolae.marasoiu@ovoenergy.com> wrote:

> Hello kafka community,
> Is there a way from kafka streams to emit a tombstone? say we have an
> events topic with upsert or delete; and an output topic with the current
> state only, log compacted; an upsert event will get mapped to current user
> topic (it holds the full entity); but a delete would perhaps need to emit a
> tombstone message (is there such a thing?) so that the “record” is
> effectively / logically deleted from the current state compacted log
> What would be the manner to achieve this using kafka-streams?
> Thank you,
>
> --
> Thank you,
> Nicolae Marasoiu
> Scala Engineer
> Orion, OVO Group
>


-- 
-- Guozhang