You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Joshua Schumacher <fr...@gmail.com> on 2015/01/29 22:12:50 UTC

Question on ETL while replau

What's the best way to add two 'fields' to my kafka messages once they are
stored?  Can I just do a replay on all of them and add the field?  How
would I throw out the old kafka messages that don't have the field then?  I
am using Druid to process the data, but not sure of how to propagate data
changes all the way down to kafka.

Thanks,
Josh

Re: Question on ETL while replau

Posted by Jun Rao <ju...@confluent.io>.

You can't change existing messages. You can republish messages with new
fields and manually set the consumer offsets.

Thanks,

Jun

On Thu, Jan 29, 2015 at 1:12 PM, Joshua Schumacher <fr...@gmail.com>
wrote:

> What's the best way to add two 'fields' to my kafka messages once they are
> stored?  Can I just do a replay on all of them and add the field?  How
> would I throw out the old kafka messages that don't have the field then?  I
> am using Druid to process the data, but not sure of how to propagate data
> changes all the way down to kafka.
>
> Thanks,
> Josh
>