You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/10/08 17:16:48 UTC
[GitHub] [iceberg] myandpr opened a new pull request #3255: [Iceberg] Flink:auto delete expired snapshots
myandpr opened a new pull request #3255:
URL: https://github.com/apache/iceberg/pull/3255
## What problem does this pr solved?
when we write data into icebeg table using flink, it will produce a lot of small files.
Merely compacting small files cannot reduce the number of small files.
Only after compacting the small files and deleting expired snapshot files can we really reduce the number of small files overall.
This branch supports deleting expired snapshots automatically when writing data into iceberg table in flink.
## How was this problem solved?
**We added the following configuration**
```
// auto expire snapshots
snapshot.flink.auto-expire.enabled
snapshot.flink.auto-expire.interval-ms
snapshot.flink.auto-expire.max-snapshot-age-ms
snapshot.flink.auto-expire.min-snapshots-to-keep
snapshot.flink.auto-expire.snapshots-group-nums
```
In this PR, we add 2 operators(ExpireSnapshotGenerator, ExpireSnapshotOperator) for expire snapshots function after IcebergFilesCommitter.
**expire snapshot function:**<br/>
1. In IcebergFilesCommitter.notifyCheckpointComplete(), we emit a 'EndCheckpoint' message to downstream operator 'ExpireSnapshotGenerator'. <br/>
2. 'ExpireSnapshotGenerator' generates all files(manifest list file, menifest file, data file) that needs to be deleted and distributes them to downstream operator 'ExpireSnapshotOperator'. <br/>
3. 'ExpireSnapshotOperator' start to actually delete all files received from upstream in parallel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3255: [Iceberg] Flink:auto delete expired snapshots
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3255:
URL: https://github.com/apache/iceberg/pull/3255#issuecomment-939527365
@myandpr, I have a few questions about the approach of this PR. First, what is the use case that motivates having separate configuration for this rather than using the existing config for snapshot expiration? I think that snapshot expiration policy and processes that actually perform the expiration are two orthogonal things. If you want to run expiration from Flink, that makes sense. But expiring snapshots differently when running in Flink vs some other method seems incorrect to me.
Second, how are you identifying the files to delete? There are two methods, incremental and reachability based. Incremental is no longer our recommended way of cleaning up files because it is really hard to reason about certain cases when snapshots are cherry-picked across branches. It isn't necessarily bad to continue using it here, but we should consider whether we can improve it. For example, I think this only cares about expiring snapshots in the current table state, which is an easier problem. Maybe we can extend the operation to support that more directly and avoid some of the hard cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org