Re: GDPR appliance

[David Espinosa <es...@gmail.com>]

Thanks a lot. I think that's the only way that ensures GDPR compliance.
In a second iteration, my thoughts are to anonymize instead of removing,
maybe identifying PII fields using AVRO custom types.

Thanks again,

2017-11-28 15:54 GMT+01:00 Ben Stopford <be...@confluent.io>:

> You should also be able to manage this with a compacted topic. If you give
> each message a unique key you'd then be able to delete, or overwrite
> specific records. Kafka will delete them from disk when compaction runs. If
> you need to partition for ordering purposes you'd need to use a custom
> partitioner that extracts a partition key from the unique key before it
> does the hash.
>
> B
>
> On Sun, Nov 26, 2017 at 10:40 AM Wim Van Leuven <
> wim.vanleuven@highestpoint.biz> wrote:
>
> > Thanks, Lars, for the most interesting read!
> >
> >
> >
> > On Sun, 26 Nov 2017 at 00:38 Lars Albertsson <la...@mapflat.com> wrote:
> >
> > > Hi David,
> > >
> > > You might find this presentation useful:
> > > https://www.slideshare.net/lallea/protecting-privacy-in-practice
> > >
> > > It explains privacy building blocks primarily in a batch processing
> > > context, but most of the principles are applicable for stream
> > > processing as well, e.g. splitting non-PII and PII data ("ejected
> > > record" slide), encrypting PII data ("lost key" slide).
> > >
> > > Regards,
> > >
> > >
> > >
> > > Lars Albertsson
> > > Data engineering consultant
> > > www.mapflat.com
> > > https://twitter.com/lalleal
> > > +46 70 7687109 <+46%2070%20768%2071%2009> <+46%2070%20768%2071%2009>
> > > Calendar: http://www.mapflat.com/calendar
> > >
> > >
> > > On Wed, Nov 22, 2017 at 7:46 PM, David Espinosa <es...@gmail.com>
> > wrote:
> > > > Hi all,
> > > > I would like to double check with you how we want to apply some GDPR
> > into
> > > > my kafka topics. In concrete the "right to be forgotten", what forces
> > us
> > > to
> > > > delete some data contained in the messages. So not deleting the
> > message,
> > > > but editing it.
> > > > For doing that, my intention is to replicate the topic and apply a
> > > > transformation over it.
> > > > I think that frameworks like Kafka Streams or Apache Storm.
> > > >
> > > > Did anybody had to solve this problem?
> > > >
> > > > Thanks in advance.
> > >
> >
>