You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@amoro.apache.org by Xavier Bai <xu...@apache.org> on 2024/09/02 08:10:08 UTC
Support for configuring the number of remaining snapshots
Hi developers,
Currently, when we execute expired snapshots, we only consider the TTL as a
factor. However, Iceberg supports setting a minimum number of snapshots to
retain. I believe we should also make this configuration option
available(eg. `snapshot.base.keep.min-count`). For developers and table
users, these snapshots can be understood as the update logs of the table,
making it easier for users to review the history of updates. If there are
no updates for an extended period, Amoro may only retain a single snapshot
after cleanup, which could result in a loss of information for users.
In addition, the Iceberg community is also working on retaining more
historical snapshot information in an additional folder. I believe this is
a significant requirement
Best regards,
Xu Bai
Re: Support for configuring the number of remaining snapshots
Posted by Jinsong Zhou <ji...@apache.org>.
Hi,
+1 for supporting configuring the number of remaining snapshots.
Besides, we may need to refactor the configuration name to:
* snapshot.keep.min-count = 10
* snapshot.keep.duration = 7d
Base is a concept of Mixed format, we should not keep it in the common
configuration names.
There is some other issue[1] focusing on improving configuration value with
duration unit.
[1] https://github.com/apache/amoro/issues/3119
Best,
Jinsong
On Mon, Sep 2, 2024 at 6:02 PM Paul Lam <pa...@gmail.com> wrote:
> +1 for supporting the snapshots to keep.
>
> However, Iceberg natively supports `history.expire.min-snapshots-to-keep`,
> should we directly reuse its value?
>
> Best,
> Paul Lam
>
> > 2024年9月2日 16:10，Xavier Bai <xu...@apache.org> 写道：
> >
> > Hi developers,
> >
> > Currently, when we execute expired snapshots, we only consider the TTL
> as a
> > factor. However, Iceberg supports setting a minimum number of snapshots
> to
> > retain. I believe we should also make this configuration option
> > available(eg. `snapshot.base.keep.min-count`). For developers and table
> > users, these snapshots can be understood as the update logs of the table,
> > making it easier for users to review the history of updates. If there are
> > no updates for an extended period, Amoro may only retain a single
> snapshot
> > after cleanup, which could result in a loss of information for users.
> >
> > In addition, the Iceberg community is also working on retaining more
> > historical snapshot information in an additional folder. I believe this
> is
> > a significant requirement
> >
> > Best regards,
> > Xu Bai
>
>
Re: Support for configuring the number of remaining snapshots
Posted by Xavier Bai <xu...@apache.org>.
I pushed a PR for this feature: https://github.com/apache/amoro/pull/3164
Jinsong Zhou <ji...@apache.org> 于2024年9月2日周一 19:18写道：
> Hi,
>
> Amoro may need some of its own configuration definitions across multiple
> table formats.
> However, the default configuration value may be taken from some native
> table format configurations such as `history.expire.min-snapshots-to-keep`.
>
> Best,
> Jinsong
>
> On Mon, Sep 2, 2024 at 6:02 PM Paul Lam <pa...@gmail.com> wrote:
>
> > +1 for supporting the snapshots to keep.
> >
> > However, Iceberg natively supports
> `history.expire.min-snapshots-to-keep`,
> > should we directly reuse its value?
> >
> > Best,
> > Paul Lam
> >
> > > 2024年9月2日 16:10，Xavier Bai <xu...@apache.org> 写道：
> > >
> > > Hi developers,
> > >
> > > Currently, when we execute expired snapshots, we only consider the TTL
> > as a
> > > factor. However, Iceberg supports setting a minimum number of snapshots
> > to
> > > retain. I believe we should also make this configuration option
> > > available(eg. `snapshot.base.keep.min-count`). For developers and table
> > > users, these snapshots can be understood as the update logs of the
> table,
> > > making it easier for users to review the history of updates. If there
> are
> > > no updates for an extended period, Amoro may only retain a single
> > snapshot
> > > after cleanup, which could result in a loss of information for users.
> > >
> > > In addition, the Iceberg community is also working on retaining more
> > > historical snapshot information in an additional folder. I believe this
> > is
> > > a significant requirement
> > >
> > > Best regards,
> > > Xu Bai
> >
> >
>
Re: Support for configuring the number of remaining snapshots
Posted by Jinsong Zhou <ji...@apache.org>.
Hi,
Amoro may need some of its own configuration definitions across multiple
table formats.
However, the default configuration value may be taken from some native
table format configurations such as `history.expire.min-snapshots-to-keep`.
Best,
Jinsong
On Mon, Sep 2, 2024 at 6:02 PM Paul Lam <pa...@gmail.com> wrote:
> +1 for supporting the snapshots to keep.
>
> However, Iceberg natively supports `history.expire.min-snapshots-to-keep`,
> should we directly reuse its value?
>
> Best,
> Paul Lam
>
> > 2024年9月2日 16:10，Xavier Bai <xu...@apache.org> 写道：
> >
> > Hi developers,
> >
> > Currently, when we execute expired snapshots, we only consider the TTL
> as a
> > factor. However, Iceberg supports setting a minimum number of snapshots
> to
> > retain. I believe we should also make this configuration option
> > available(eg. `snapshot.base.keep.min-count`). For developers and table
> > users, these snapshots can be understood as the update logs of the table,
> > making it easier for users to review the history of updates. If there are
> > no updates for an extended period, Amoro may only retain a single
> snapshot
> > after cleanup, which could result in a loss of information for users.
> >
> > In addition, the Iceberg community is also working on retaining more
> > historical snapshot information in an additional folder. I believe this
> is
> > a significant requirement
> >
> > Best regards,
> > Xu Bai
>
>
Re: Support for configuring the number of remaining snapshots
Posted by Paul Lam <pa...@gmail.com>.
+1 for supporting the snapshots to keep.
However, Iceberg natively supports `history.expire.min-snapshots-to-keep`, should we directly reuse its value?
Best,
Paul Lam
> 2024年9月2日 16:10，Xavier Bai <xu...@apache.org> 写道：
>
> Hi developers,
>
> Currently, when we execute expired snapshots, we only consider the TTL as a
> factor. However, Iceberg supports setting a minimum number of snapshots to
> retain. I believe we should also make this configuration option
> available(eg. `snapshot.base.keep.min-count`). For developers and table
> users, these snapshots can be understood as the update logs of the table,
> making it easier for users to review the history of updates. If there are
> no updates for an extended period, Amoro may only retain a single snapshot
> after cleanup, which could result in a loss of information for users.
>
> In addition, the Iceberg community is also working on retaining more
> historical snapshot information in an additional folder. I believe this is
> a significant requirement
>
> Best regards,
> Xu Bai