You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Максим Радионов <ra...@gmail.com> on 2019/08/26 08:06:03 UTC
HoodieDeltaStreamer commits history
Hi all,
I use COW and HoodieDeltaStreamer and after successful execution of a new
job I would like for the old data with commits to be deleted and there
would be a data snapshot without the history of commits, please tell me the
correct way to solve this task.
example: (i whould like to delete old file commit after succesed job)
fe57cbbe-fea9-493d-8c20-b74c5b699597_200_20190803114918.parquet
fe57cbbe-fea9-493d-8c20-b74c5b699597_1479_20190804125520.parquet
Kind regards,
Maksim
Re: HoodieDeltaStreamer commits history
Posted by Vinoth Chandar <vi...@apache.org>.
+1 One caveat here is that long queries (e.g Hive) may still be accessing
these older files and may fail when cleaning very aggressively like this.
On Mon, Aug 26, 2019 at 2:24 AM Gary Li <ya...@gmail.com> wrote:
> Hello,
>
> You can achieve this by changing the hudi config.
> hoodie.cleaner.policy=KEEP_LATEST_FILE_VERSIONS
> hoodie.cleaner.commits.retained=1
>
> Best,
> Gary
>
>
> On Mon, Aug 26, 2019 at 10:06 AM Максим Радионов <radionov.maxim@gmail.com
> >
> wrote:
>
> > Hi all,
> >
> > I use COW and HoodieDeltaStreamer and after successful execution of a new
> > job I would like for the old data with commits to be deleted and there
> > would be a data snapshot without the history of commits, please tell me
> the
> > correct way to solve this task.
> >
> > example: (i whould like to delete old file commit after succesed job)
> > fe57cbbe-fea9-493d-8c20-b74c5b699597_200_20190803114918.parquet
> > fe57cbbe-fea9-493d-8c20-b74c5b699597_1479_20190804125520.parquet
> >
> > Kind regards,
> > Maksim
> >
>
Re: HoodieDeltaStreamer commits history
Posted by Gary Li <ya...@gmail.com>.
Hello,
You can achieve this by changing the hudi config.
hoodie.cleaner.policy=KEEP_LATEST_FILE_VERSIONS
hoodie.cleaner.commits.retained=1
Best,
Gary
On Mon, Aug 26, 2019 at 10:06 AM Максим Радионов <ra...@gmail.com>
wrote:
> Hi all,
>
> I use COW and HoodieDeltaStreamer and after successful execution of a new
> job I would like for the old data with commits to be deleted and there
> would be a data snapshot without the history of commits, please tell me the
> correct way to solve this task.
>
> example: (i whould like to delete old file commit after succesed job)
> fe57cbbe-fea9-493d-8c20-b74c5b699597_200_20190803114918.parquet
> fe57cbbe-fea9-493d-8c20-b74c5b699597_1479_20190804125520.parquet
>
> Kind regards,
> Maksim
>