You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hudi.apache.org by Li Gao <li...@gmail.com> on 2019/04/23 23:12:25 UTC

Hudi support for records deduplication

Hi Hudi community,

I am fairly new to the hudi community and trying to evaluate whether hudi's
incremental compaction support record deduplications from landing data
coming off kafka. If yes I want to understand how it works currently.

Thank you,
Li

Re: Hudi support for records deduplication

Posted by Vinoth Chandar <vi...@apache.org>.

Hi Li,

Welcome. Both the delta streamer and data source support an option to
de-duplicate data before inserting.
How are you planning on writing the Hudi dataset? I can point you in the
right direction accordingly

Thanks
Vinoth

On Tue, Apr 23, 2019 at 4:12 PM Li Gao <li...@gmail.com> wrote:

> Hi Hudi community,
>
> I am fairly new to the hudi community and trying to evaluate whether hudi's
> incremental compaction support record deduplications from landing data
> coming off kafka. If yes I want to understand how it works currently.
>
> Thank you,
> Li
>