You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "shangxinli (via GitHub)" <gi...@apache.org> on 2026/03/16 03:33:38 UTC
[PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
shangxinli opened a new pull request, #592:
URL: https://github.com/apache/iceberg-cpp/pull/592
## Summary
- Implement the file cleanup logic missing from expire snapshots (#490 noted "TODO: File recycling will be added in a followup PR")
- Port the "reachable file cleanup" strategy from Java's `ReachableFileCleanup`
- Single-threaded implementation; multi-threaded and incremental cleanup as TODOs
## Changes
- Add `Finalize()` override called after successful commit to clean up expired files
- Add `CleanExpiredFiles()` implementing the reachable file cleanup strategy:
1. Collect manifest paths from expired and retained snapshots
2. Prune manifests still referenced by retained snapshots
3. Find data files only in manifests being deleted, subtract files still reachable from retained manifests
4. Delete orphaned manifests, manifest lists, and statistics files
- Best-effort deletion: suppress errors on individual file deletions to avoid blocking metadata updates (matching Java's `suppressFailureWhenFinished`)
- Branch/tag awareness: retained snapshot set includes all snapshots reachable from any ref
- Respect `CleanupLevel`: `kNone` skips all, `kMetadataOnly` skips data files, `kAll` cleans everything
- Uses `FileIO::DeleteFile` for filesystem compatibility (S3, HDFS, local)
- 5 new tests for file cleanup behavior
## Test plan
- [x] All 303 existing tests pass
- [x] 9 expire snapshots tests pass (4 existing + 5 new)
- [x] `CleanupLevelNoneSkipsFileDeletion` — verifies kNone skips all deletion
- [x] `FinalizeSkippedOnCommitError` — verifies no cleanup on commit failure
- [x] `FinalizeSkippedWhenNoSnapshotsExpired` — verifies no cleanup when nothing expired
- [x] `DeleteWithCustomFunction` — verifies custom delete function is invoked
- [x] `CommitWithCleanupLevelNone` — end-to-end commit with metadata update
Closes #364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r2965627717
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.insert(entry.data_file->file_path);
+ }
+ }
+ goto next_manifest; // Found and processed this manifest, move to next
+ }
+ }
+ next_manifest:;
+ }
+
+ if (data_files_to_delete.empty()) {
+ return {};
+ }
+
+ // Step 2: Remove any files that are still referenced by retained manifests.
+ // This ensures we don't delete files that are shared across manifests.
+ for (const auto& manifest_path : retained_manifests) {
+ if (data_files_to_delete.empty()) break;
+
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
Review Comment:
**Inconsistent Behavior (Data Loss Risk):** If reading a **retained** manifest fails, the C++ implementation silently ignores it (`continue`). Java uses `.retry(3)` and `.throwFailureWhenFinished()`. If we silently ignore a read failure here, we will fail to subtract its live data files from `data_files_to_delete`, resulting in **accidental data loss** (deleting a physical file that is still actively used). Failures here should abort the deletion of those specific data files.
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.insert(entry.data_file->file_path);
+ }
+ }
+ goto next_manifest; // Found and processed this manifest, move to next
Review Comment:
**C++ Style Issue:** The use of `goto next_manifest;` to break out of nested loops is a non-idiomatic C++ anti-pattern. Consider moving this manifest lookup logic into a helper function (e.g., `std::optional<ManifestFile> GetManifestByPath(path)`).
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
Review Comment:
**C++ Style Issue:** Using mutable out-parameters (`std::unordered_set<std::string>& data_files_to_delete`) over `Result` is a less modern C++ pattern. Consider returning `Result<std::unordered_set<std::string>>`.
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.insert(entry.data_file->file_path);
+ }
+ }
+ goto next_manifest; // Found and processed this manifest, move to next
+ }
+ }
+ next_manifest:;
+ }
+
+ if (data_files_to_delete.empty()) {
+ return {};
+ }
+
+ // Step 2: Remove any files that are still referenced by retained manifests.
+ // This ensures we don't delete files that are shared across manifests.
+ for (const auto& manifest_path : retained_manifests) {
+ if (data_files_to_delete.empty()) break;
+
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.erase(entry.data_file->file_path);
+ }
+ }
+ goto next_retained;
+ }
+ }
+ next_retained:;
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::CleanExpiredFiles(
+ const std::vector<int64_t>& expired_snapshot_ids) {
+ const TableMetadata& metadata = base();
+
+ // Build expired and retained snapshot ID sets.
+ // The retained set includes ALL snapshots referenced by any branch or tag,
+ // since Apply() already computed retention across all refs.
+ std::unordered_set<int64_t> expired_id_set(expired_snapshot_ids.begin(),
+ expired_snapshot_ids.end());
+ std::unordered_set<int64_t> retained_snapshot_ids;
+ for (const auto& snapshot : metadata.snapshots) {
+ if (snapshot && !expired_id_set.contains(snapshot->snapshot_id)) {
+ retained_snapshot_ids.insert(snapshot->snapshot_id);
+ }
+ }
+
+ // Phase 1: Collect manifest paths from expired and retained snapshots.
+ // TODO(shangxinli): Parallelize manifest collection with a thread pool.
+ std::unordered_set<std::string> expired_manifest_paths;
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ std::ignore = ReadManifestsForSnapshot(snapshot_id, expired_manifest_paths);
+ }
+
+ std::unordered_set<std::string> retained_manifest_paths;
+ for (int64_t snapshot_id : retained_snapshot_ids) {
+ std::ignore = ReadManifestsForSnapshot(snapshot_id, retained_manifest_paths);
+ }
+
+ // Phase 2: Prune manifests still referenced by retained snapshots.
+ // Only manifests exclusively in expired snapshots should be deleted.
+ std::unordered_set<std::string> manifests_to_delete;
+ for (const auto& path : expired_manifest_paths) {
+ if (!retained_manifest_paths.contains(path)) {
+ manifests_to_delete.insert(path);
+ }
+ }
+
+ // Phase 3: If cleanup level is kAll, find data files to delete.
+ // Only read entries from manifests being deleted (not all expired manifests),
+ // then subtract any files still reachable from retained manifests.
+ if (cleanup_level_ == CleanupLevel::kAll && !manifests_to_delete.empty()) {
+ std::unordered_set<std::string> data_files_to_delete;
+ std::ignore = FindDataFilesToDelete(manifests_to_delete, retained_manifest_paths,
+ data_files_to_delete);
+
+ // TODO(shangxinli): Parallelize file deletion with a thread pool.
+ for (const auto& path : data_files_to_delete) {
+ DeleteFilePath(path);
+ }
+ }
+
+ // Phase 4: Delete orphaned manifest files.
+ for (const auto& path : manifests_to_delete) {
+ DeleteFilePath(path);
+ }
+
+ // Phase 5: Delete manifest lists from expired snapshots.
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) continue;
+ auto& snapshot = snapshot_result.value();
+ if (!snapshot->manifest_list.empty()) {
+ DeleteFilePath(snapshot->manifest_list);
+ }
+ }
+
+ // Phase 6: Delete expired statistics files.
+ // Use set difference between before and after states (matching Java behavior).
+ // Since Finalize runs before table_ is updated, "after" is base() minus expired.
+ std::unordered_set<int64_t> retained_stats_snapshots(retained_snapshot_ids);
+ for (const auto& stat_file : metadata.statistics) {
+ if (stat_file && !retained_stats_snapshots.contains(stat_file->snapshot_id)) {
Review Comment:
**Inconsistent Behavior:** C++ deletes statistics files by checking if the `snapshot_id` of the `StatisticsFile` is in the `retained_stats_snapshots` set. Java computes the set difference of actual **file paths**. If a statistics file path is referenced by multiple snapshots, deleting purely based on the expiration of a specific `snapshot_id` could erroneously delete a physical file that is still referenced by a newer, retained snapshot. Diff paths, not just snapshot IDs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r2965600427
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.insert(entry.data_file->file_path);
+ }
+ }
+ goto next_manifest; // Found and processed this manifest, move to next
+ }
+ }
+ next_manifest:;
+ }
+
+ if (data_files_to_delete.empty()) {
+ return {};
+ }
+
+ // Step 2: Remove any files that are still referenced by retained manifests.
+ // This ensures we don't delete files that are shared across manifests.
+ for (const auto& manifest_path : retained_manifests) {
+ if (data_files_to_delete.empty()) break;
+
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
Review Comment:
**Parity / Logic Issue:** Similar to line 387, use `LiveEntries()` here instead of `Entries()` to avoid a potential storage leak.
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.insert(entry.data_file->file_path);
+ }
+ }
+ goto next_manifest; // Found and processed this manifest, move to next
+ }
+ }
+ next_manifest:;
+ }
+
+ if (data_files_to_delete.empty()) {
+ return {};
+ }
+
+ // Step 2: Remove any files that are still referenced by retained manifests.
+ // This ensures we don't delete files that are shared across manifests.
+ for (const auto& manifest_path : retained_manifests) {
+ if (data_files_to_delete.empty()) break;
+
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
Review Comment:
**Performance / Logic Issue:** Similar to line 371, instantiating `SnapshotCache` here inside the loop repeats the O(M*S) I/O complexity.
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
Review Comment:
**Performance / Logic Issue:** Instantiating `SnapshotCache` locally inside the nested loop causes O(M*S) I/O complexity. The cache is not preserved across loop iterations, meaning for *every* manifest path being processed, the manifest list of *every* snapshot is repeatedly downloaded and parsed from storage.
*Suggestion:* Consider collecting `ManifestFile` objects in Phase 1 (similar to Java's `Set<ManifestFile> deletionCandidates`) or pre-caching them in a map to avoid redundant I/O.
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
Review Comment:
**Parity / Logic Issue:** C++ uses `Entries()` which includes `DELETED` entries, whereas Java uses `ManifestFiles.readPaths` which delegates to `liveEntries()` (only `ADDED` and `EXISTING`).
*Details:* If a data file is added in an expired snapshot and deleted in a retained snapshot, using `Entries()` will read the DELETED entry in Phase 2 and subtract the data file from `data_files_to_delete`. This prevents the physical file from being deleted, resulting in a storage leak.
*Suggestion:* Change `reader_result.value()->Entries()` to `reader_result.value()->LiveEntries()` to strictly match Java's behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#issuecomment-4163252655
@wgtmac made the change. Can you have a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r3035741060
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,250 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Result<std::shared_ptr<ManifestReader>> ExpireSnapshots::MakeManifestReader(
+ const ManifestFile& manifest, const std::shared_ptr<FileIO>& file_io) {
+ const TableMetadata& metadata = base();
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) return std::unexpected<Error>(schema_result.error());
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) return std::unexpected<Error>(spec_result.error());
+ return ManifestReader::Make(manifest, file_io, schema_result.value(),
+ spec_result.value());
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ // Cache manifest metadata for later use in FindDataFilesToDelete,
+ // avoiding O(M*S) repeated I/O from re-reading manifest lists.
+ manifest_cache_.emplace(manifest.manifest_path, manifest);
+ }
+
+ return {};
+}
+
+Result<std::unordered_set<std::string>> ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests) {
+ auto file_io = ctx_->table->io();
+ std::unordered_set<std::string> data_files_to_delete;
+
+ // Step 1: Collect live file paths from manifests being deleted.
+ // Use LiveEntries() (ADDED/EXISTING only) to match Java's ManifestFiles.readPaths
+ // which delegates to liveEntries(). Using Entries() would include DELETED entries
+ // and could cause storage leaks.
+ for (const auto& [path, manifest] : manifest_cache_) {
+ if (!manifests_to_delete.contains(path)) continue;
+
+ auto reader_result = MakeManifestReader(manifest, file_io);
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->LiveEntries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.insert(entry.data_file->file_path);
+ }
+ }
+ }
+
+ if (data_files_to_delete.empty()) {
+ return data_files_to_delete;
+ }
+
+ // Step 2: Remove any files that are still referenced by retained manifests.
+ // If reading a retained manifest fails, we must NOT delete its data files
+ // to avoid accidental data loss (matching Java's retry + throwFailureWhenFinished).
+ for (const auto& manifest_path : retained_manifests) {
+ if (data_files_to_delete.empty()) break;
+
+ auto it = manifest_cache_.find(manifest_path);
+ if (it == manifest_cache_.end()) continue;
+
+ auto reader_result = MakeManifestReader(it->second, file_io);
+ if (!reader_result.has_value()) {
+ // Cannot read a retained manifest — abort data file deletion to prevent
+ // accidental data loss. Java retries and throws on failure here.
+ return std::unordered_set<std::string>{};
+ }
+
+ auto entries_result = reader_result.value()->LiveEntries();
+ if (!entries_result.has_value()) {
+ return std::unordered_set<std::string>{};
+ }
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.erase(entry.data_file->file_path);
+ }
+ }
+ }
+
+ return data_files_to_delete;
+}
+
+Status ExpireSnapshots::CleanExpiredFiles(
+ const std::vector<int64_t>& expired_snapshot_ids) {
+ const TableMetadata& metadata = base();
+
+ // Build expired and retained snapshot ID sets.
+ // The retained set includes ALL snapshots referenced by any branch or tag,
+ // since Apply() already computed retention across all refs.
+ std::unordered_set<int64_t> expired_id_set(expired_snapshot_ids.begin(),
+ expired_snapshot_ids.end());
+ std::unordered_set<int64_t> retained_snapshot_ids;
+ for (const auto& snapshot : metadata.snapshots) {
+ if (snapshot && !expired_id_set.contains(snapshot->snapshot_id)) {
+ retained_snapshot_ids.insert(snapshot->snapshot_id);
+ }
+ }
+
+ // Phase 1: Collect manifest paths from expired and retained snapshots.
+ // TODO(shangxinli): Parallelize manifest collection with a thread pool.
+ std::unordered_set<std::string> expired_manifest_paths;
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ std::ignore = ReadManifestsForSnapshot(snapshot_id, expired_manifest_paths);
Review Comment:
ReadManifestsForSnapshot now returns void since it is intentionally best-effort. For expired snapshots, a read failure means we might miss some file deletions, but orphaned files can be cleaned up by GC later. This is the conservative direction (never delete what we can't confirm is unreachable). For retained manifests in FindDataFilesToDelete, failures still abort data file deletion entirely to prevent accidental data loss, matching Java's throwFailureWhenFinished behavior there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r3035738291
##########
src/iceberg/update/expire_snapshots.h:
##########
@@ -159,6 +173,47 @@ class ICEBERG_EXPORT ExpireSnapshots : public PendingUpdate {
Result<std::unordered_set<int64_t>> UnreferencedSnapshotIdsToRetain(
const SnapshotToRef& refs) const;
+ /// \brief Clean up files no longer referenced after snapshot expiration.
+ ///
+ /// Implements the "reachable file cleanup" strategy from Java's ReachableFileCleanup:
+ /// 1. Collect manifests from expired and retained snapshots
+ /// 2. Prune manifests still referenced by retained snapshots
+ /// 3. Find data files only in manifests being deleted (if kAll)
+ /// 4. Remove data files still reachable from retained manifests
+ /// 5. Delete orphaned manifests, manifest lists, and statistics files
+ ///
+ /// All deletions are best-effort: failures are suppressed to avoid blocking
+ /// metadata updates (matching Java's suppressFailureWhenFinished behavior).
+ ///
+ /// Branch/tag awareness: retained_snapshot_ids includes all snapshots referenced
+ /// by any branch or tag, as computed by Apply(). This prevents deleting files
+ /// that are still reachable from any ref.
+ ///
+ /// TODO(shangxinli): Add multi-threaded file deletion support.
+ /// TODO(shangxinli): Add IncrementalFileCleanup strategy for linear ancestry.
+ Status CleanExpiredFiles(const std::vector<int64_t>& expired_snapshot_ids);
+
+ /// \brief Read manifest paths from a single snapshot.
+ /// Best-effort: returns OK even if the snapshot or its manifests can't be read.
+ Status ReadManifestsForSnapshot(int64_t snapshot_id,
Review Comment:
Done — introduced FileCleanupStrategy abstract base class and ReachableFileCleanup concrete subclass in the anonymous namespace of expire_snapshots.cc, mirroring Java's class hierarchy. They live in the .cc for now since they're unlikely to be used elsewhere; we can promote to a header when IncrementalFileCleanup is added.
##########
src/iceberg/update/expire_snapshots.h:
##########
@@ -159,6 +173,47 @@ class ICEBERG_EXPORT ExpireSnapshots : public PendingUpdate {
Result<std::unordered_set<int64_t>> UnreferencedSnapshotIdsToRetain(
const SnapshotToRef& refs) const;
+ /// \brief Clean up files no longer referenced after snapshot expiration.
+ ///
+ /// Implements the "reachable file cleanup" strategy from Java's ReachableFileCleanup:
+ /// 1. Collect manifests from expired and retained snapshots
+ /// 2. Prune manifests still referenced by retained snapshots
+ /// 3. Find data files only in manifests being deleted (if kAll)
+ /// 4. Remove data files still reachable from retained manifests
+ /// 5. Delete orphaned manifests, manifest lists, and statistics files
+ ///
+ /// All deletions are best-effort: failures are suppressed to avoid blocking
+ /// metadata updates (matching Java's suppressFailureWhenFinished behavior).
+ ///
+ /// Branch/tag awareness: retained_snapshot_ids includes all snapshots referenced
+ /// by any branch or tag, as computed by Apply(). This prevents deleting files
+ /// that are still reachable from any ref.
+ ///
+ /// TODO(shangxinli): Add multi-threaded file deletion support.
+ /// TODO(shangxinli): Add IncrementalFileCleanup strategy for linear ancestry.
+ Status CleanExpiredFiles(const std::vector<int64_t>& expired_snapshot_ids);
+
+ /// \brief Read manifest paths from a single snapshot.
+ /// Best-effort: returns OK even if the snapshot or its manifests can't be read.
+ Status ReadManifestsForSnapshot(int64_t snapshot_id,
+ std::unordered_set<std::string>& manifest_paths);
+
+ /// \brief Find data files to delete by reading live entries from manifests being
+ /// deleted, then subtracting files still reachable from retained manifests.
+ /// If a retained manifest cannot be read, returns an empty set to prevent
+ /// accidental data loss.
+ Result<std::unordered_set<std::string>> FindDataFilesToDelete(
Review Comment:
Done — ReadManifestsForSnapshot is now a private method of ReachableFileCleanup only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r3035738693
##########
src/iceberg/update/expire_snapshots.h:
##########
@@ -159,6 +173,47 @@ class ICEBERG_EXPORT ExpireSnapshots : public PendingUpdate {
Result<std::unordered_set<int64_t>> UnreferencedSnapshotIdsToRetain(
const SnapshotToRef& refs) const;
+ /// \brief Clean up files no longer referenced after snapshot expiration.
+ ///
+ /// Implements the "reachable file cleanup" strategy from Java's ReachableFileCleanup:
+ /// 1. Collect manifests from expired and retained snapshots
+ /// 2. Prune manifests still referenced by retained snapshots
+ /// 3. Find data files only in manifests being deleted (if kAll)
+ /// 4. Remove data files still reachable from retained manifests
+ /// 5. Delete orphaned manifests, manifest lists, and statistics files
+ ///
+ /// All deletions are best-effort: failures are suppressed to avoid blocking
+ /// metadata updates (matching Java's suppressFailureWhenFinished behavior).
+ ///
+ /// Branch/tag awareness: retained_snapshot_ids includes all snapshots referenced
+ /// by any branch or tag, as computed by Apply(). This prevents deleting files
+ /// that are still reachable from any ref.
+ ///
+ /// TODO(shangxinli): Add multi-threaded file deletion support.
+ /// TODO(shangxinli): Add IncrementalFileCleanup strategy for linear ancestry.
+ Status CleanExpiredFiles(const std::vector<int64_t>& expired_snapshot_ids);
+
+ /// \brief Read manifest paths from a single snapshot.
+ /// Best-effort: returns OK even if the snapshot or its manifests can't be read.
+ Status ReadManifestsForSnapshot(int64_t snapshot_id,
+ std::unordered_set<std::string>& manifest_paths);
+
+ /// \brief Find data files to delete by reading live entries from manifests being
+ /// deleted, then subtracting files still reachable from retained manifests.
+ /// If a retained manifest cannot be read, returns an empty set to prevent
+ /// accidental data loss.
+ Result<std::unordered_set<std::string>> FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests);
+
+ /// \brief Create a ManifestReader for the given ManifestFile.
+ Result<std::shared_ptr<ManifestReader>> MakeManifestReader(
Review Comment:
Done — MakeManifestReader is now a free function in the anonymous namespace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: add ReachableFileCleanup to expire snapshots [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r3099347039
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -37,6 +43,264 @@
namespace iceberg {
+namespace {
+
+Result<std::shared_ptr<ManifestReader>> MakeManifestReader(
+ const ManifestFile& manifest, const std::shared_ptr<FileIO>& file_io,
+ const TableMetadata& metadata) {
+ ICEBERG_ASSIGN_OR_RAISE(auto schema, metadata.Schema());
+ ICEBERG_ASSIGN_OR_RAISE(auto spec,
+ metadata.PartitionSpecById(manifest.partition_spec_id));
+ return ManifestReader::Make(manifest, file_io, std::move(schema), std::move(spec));
+}
+
+/// \brief Abstract strategy for cleaning up files after snapshot expiration.
+///
+/// Mirrors Java's FileCleanupStrategy: provides shared delete utilities while
+/// allowing different cleanup algorithms (ReachableFileCleanup, IncrementalFileCleanup).
+class FileCleanupStrategy {
+ public:
+ FileCleanupStrategy(std::shared_ptr<FileIO> file_io,
+ std::function<void(const std::string&)> delete_func)
+ : file_io_(std::move(file_io)), delete_func_(std::move(delete_func)) {}
+
+ virtual ~FileCleanupStrategy() = default;
+
+ /// \brief Clean up files that are only reachable by expired snapshots.
+ ///
+ /// \param metadata Table metadata before expiration (contains all snapshots).
+ /// \param expired_snapshot_ids Snapshot IDs that were expired during this operation.
+ /// \param level Controls which types of files are eligible for deletion.
+ virtual Status CleanFiles(const TableMetadata& metadata,
+ const std::unordered_set<int64_t>& expired_snapshot_ids,
+ CleanupLevel level) = 0;
+
+ protected:
+ /// \brief Delete a file, suppressing errors (best-effort).
+ ///
+ /// Uses the custom delete function if set, otherwise FileIO::DeleteFile.
+ /// Matches Java's suppressFailureWhenFinished behavior.
+ void DeleteFile(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ std::ignore = file_io_->DeleteFile(path);
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+ }
+
+ /// \brief Returns paths of statistics files referenced only by expired snapshots.
+ ///
+ /// Uses path-based set difference (matching Java's expiredStatisticsFilesLocations):
+ /// if the same file path is shared across snapshots, it is only deleted when
+ /// no retained snapshot references it.
+ std::unordered_set<std::string> ExpiredStatisticsFilePaths(
+ const TableMetadata& metadata, const std::unordered_set<int64_t>& expired_ids) {
+ std::unordered_set<std::string> retained_paths;
+ for (const auto& stat : metadata.statistics) {
+ if (stat && !expired_ids.contains(stat->snapshot_id)) {
+ retained_paths.insert(stat->path);
+ }
+ }
+ for (const auto& part_stat : metadata.partition_statistics) {
+ if (part_stat && !expired_ids.contains(part_stat->snapshot_id)) {
+ retained_paths.insert(part_stat->path);
+ }
+ }
+
+ std::unordered_set<std::string> expired_paths;
+ for (const auto& stat : metadata.statistics) {
+ if (stat && expired_ids.contains(stat->snapshot_id) &&
+ !retained_paths.contains(stat->path)) {
+ expired_paths.insert(stat->path);
+ }
+ }
+ for (const auto& part_stat : metadata.partition_statistics) {
+ if (part_stat && expired_ids.contains(part_stat->snapshot_id) &&
+ !retained_paths.contains(part_stat->path)) {
+ expired_paths.insert(part_stat->path);
+ }
+ }
+ return expired_paths;
+ }
+
+ std::shared_ptr<FileIO> file_io_;
+ std::function<void(const std::string&)> delete_func_;
+};
+
+/// \brief File cleanup strategy that determines safe deletions via full reachability.
+///
+/// Mirrors Java's ReachableFileCleanup: collects manifests from all expired and
+/// retained snapshots, prunes candidates still referenced by retained snapshots,
+/// then deletes orphaned manifests, data files, manifest lists, and statistics files.
+///
+/// TODO(shangxinli): Add multi-threaded manifest reading and file deletion support.
+class ReachableFileCleanup : public FileCleanupStrategy {
+ public:
+ using FileCleanupStrategy::FileCleanupStrategy;
+
+ Status CleanFiles(const TableMetadata& metadata,
+ const std::unordered_set<int64_t>& expired_snapshot_ids,
+ CleanupLevel level) override {
+ std::unordered_set<int64_t> retained_snapshot_ids;
+ for (const auto& snapshot : metadata.snapshots) {
+ if (snapshot && !expired_snapshot_ids.contains(snapshot->snapshot_id)) {
+ retained_snapshot_ids.insert(snapshot->snapshot_id);
+ }
+ }
+
+ // Phase 1: Collect manifest paths from expired and retained snapshots.
+ // The manifest_cache_ is populated here to avoid O(M*S) repeated I/O in
+ // FindDataFilesToDelete.
+ std::unordered_set<std::string> expired_manifest_paths;
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ ReadManifestsForSnapshot(metadata, snapshot_id, expired_manifest_paths);
+ }
+ std::unordered_set<std::string> retained_manifest_paths;
+ for (int64_t snapshot_id : retained_snapshot_ids) {
+ ReadManifestsForSnapshot(metadata, snapshot_id, retained_manifest_paths);
+ }
+
+ // Phase 2: Prune manifests still referenced by retained snapshots.
+ std::unordered_set<std::string> manifests_to_delete;
+ for (const auto& path : expired_manifest_paths) {
+ if (!retained_manifest_paths.contains(path)) {
+ manifests_to_delete.insert(path);
+ }
+ }
+
+ // Phase 3: Delete data files if cleanup level is kAll.
+ if (level == CleanupLevel::kAll && !manifests_to_delete.empty()) {
+ auto data_files_result =
+ FindDataFilesToDelete(metadata, manifests_to_delete, retained_manifest_paths);
+ if (data_files_result.has_value()) {
+ for (const auto& path : data_files_result.value()) {
+ DeleteFile(path);
+ }
+ }
+ }
+
+ // Phase 4: Delete orphaned manifest files.
+ for (const auto& path : manifests_to_delete) {
+ DeleteFile(path);
+ }
+
+ // Phase 5: Delete manifest lists from expired snapshots.
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) continue;
+ const auto& snapshot = snapshot_result.value();
+ if (!snapshot->manifest_list.empty()) {
+ DeleteFile(snapshot->manifest_list);
+ }
+ }
+
+ // Phase 6: Delete expired statistics files using path-based set difference.
Review Comment:
[P1] Stats files are deleted even though snapshot removal still leaves their metadata entries behind. This PR now deletes expired statistics and partition-statistics files in `Finalize`, but C++ snapshot removal still does not remove the corresponding `statistics` / `partition_statistics` metadata entries (`src/iceberg/table_metadata.cc` still has a FIXME instead of calling `RemoveStatistics` / `RemovePartitionStatistics`). That means a table can commit metadata that still references a stats file and then immediately delete the file on disk. Java removes those metadata entries as part of snapshot removal and has regression coverage for expiration and reuse of stats files. Please either wire snapshot removal to drop the metadata entries first or defer physical stats-file deletion until that is implemented.
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -37,6 +43,264 @@
namespace iceberg {
+namespace {
+
+Result<std::shared_ptr<ManifestReader>> MakeManifestReader(
+ const ManifestFile& manifest, const std::shared_ptr<FileIO>& file_io,
+ const TableMetadata& metadata) {
+ ICEBERG_ASSIGN_OR_RAISE(auto schema, metadata.Schema());
+ ICEBERG_ASSIGN_OR_RAISE(auto spec,
+ metadata.PartitionSpecById(manifest.partition_spec_id));
+ return ManifestReader::Make(manifest, file_io, std::move(schema), std::move(spec));
+}
+
+/// \brief Abstract strategy for cleaning up files after snapshot expiration.
+///
+/// Mirrors Java's FileCleanupStrategy: provides shared delete utilities while
+/// allowing different cleanup algorithms (ReachableFileCleanup, IncrementalFileCleanup).
+class FileCleanupStrategy {
+ public:
+ FileCleanupStrategy(std::shared_ptr<FileIO> file_io,
+ std::function<void(const std::string&)> delete_func)
+ : file_io_(std::move(file_io)), delete_func_(std::move(delete_func)) {}
+
+ virtual ~FileCleanupStrategy() = default;
+
+ /// \brief Clean up files that are only reachable by expired snapshots.
+ ///
+ /// \param metadata Table metadata before expiration (contains all snapshots).
+ /// \param expired_snapshot_ids Snapshot IDs that were expired during this operation.
+ /// \param level Controls which types of files are eligible for deletion.
+ virtual Status CleanFiles(const TableMetadata& metadata,
+ const std::unordered_set<int64_t>& expired_snapshot_ids,
+ CleanupLevel level) = 0;
+
+ protected:
+ /// \brief Delete a file, suppressing errors (best-effort).
+ ///
+ /// Uses the custom delete function if set, otherwise FileIO::DeleteFile.
+ /// Matches Java's suppressFailureWhenFinished behavior.
+ void DeleteFile(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ std::ignore = file_io_->DeleteFile(path);
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+ }
+
+ /// \brief Returns paths of statistics files referenced only by expired snapshots.
+ ///
+ /// Uses path-based set difference (matching Java's expiredStatisticsFilesLocations):
+ /// if the same file path is shared across snapshots, it is only deleted when
+ /// no retained snapshot references it.
+ std::unordered_set<std::string> ExpiredStatisticsFilePaths(
+ const TableMetadata& metadata, const std::unordered_set<int64_t>& expired_ids) {
+ std::unordered_set<std::string> retained_paths;
+ for (const auto& stat : metadata.statistics) {
+ if (stat && !expired_ids.contains(stat->snapshot_id)) {
+ retained_paths.insert(stat->path);
+ }
+ }
+ for (const auto& part_stat : metadata.partition_statistics) {
+ if (part_stat && !expired_ids.contains(part_stat->snapshot_id)) {
+ retained_paths.insert(part_stat->path);
+ }
+ }
+
+ std::unordered_set<std::string> expired_paths;
+ for (const auto& stat : metadata.statistics) {
+ if (stat && expired_ids.contains(stat->snapshot_id) &&
+ !retained_paths.contains(stat->path)) {
+ expired_paths.insert(stat->path);
+ }
+ }
+ for (const auto& part_stat : metadata.partition_statistics) {
+ if (part_stat && expired_ids.contains(part_stat->snapshot_id) &&
+ !retained_paths.contains(part_stat->path)) {
+ expired_paths.insert(part_stat->path);
+ }
+ }
+ return expired_paths;
+ }
+
+ std::shared_ptr<FileIO> file_io_;
+ std::function<void(const std::string&)> delete_func_;
+};
+
+/// \brief File cleanup strategy that determines safe deletions via full reachability.
+///
+/// Mirrors Java's ReachableFileCleanup: collects manifests from all expired and
+/// retained snapshots, prunes candidates still referenced by retained snapshots,
+/// then deletes orphaned manifests, data files, manifest lists, and statistics files.
+///
+/// TODO(shangxinli): Add multi-threaded manifest reading and file deletion support.
+class ReachableFileCleanup : public FileCleanupStrategy {
+ public:
+ using FileCleanupStrategy::FileCleanupStrategy;
+
+ Status CleanFiles(const TableMetadata& metadata,
+ const std::unordered_set<int64_t>& expired_snapshot_ids,
+ CleanupLevel level) override {
+ std::unordered_set<int64_t> retained_snapshot_ids;
+ for (const auto& snapshot : metadata.snapshots) {
+ if (snapshot && !expired_snapshot_ids.contains(snapshot->snapshot_id)) {
+ retained_snapshot_ids.insert(snapshot->snapshot_id);
+ }
+ }
+
+ // Phase 1: Collect manifest paths from expired and retained snapshots.
+ // The manifest_cache_ is populated here to avoid O(M*S) repeated I/O in
+ // FindDataFilesToDelete.
+ std::unordered_set<std::string> expired_manifest_paths;
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ ReadManifestsForSnapshot(metadata, snapshot_id, expired_manifest_paths);
+ }
+ std::unordered_set<std::string> retained_manifest_paths;
+ for (int64_t snapshot_id : retained_snapshot_ids) {
+ ReadManifestsForSnapshot(metadata, snapshot_id, retained_manifest_paths);
+ }
+
+ // Phase 2: Prune manifests still referenced by retained snapshots.
+ std::unordered_set<std::string> manifests_to_delete;
+ for (const auto& path : expired_manifest_paths) {
+ if (!retained_manifest_paths.contains(path)) {
+ manifests_to_delete.insert(path);
+ }
+ }
+
+ // Phase 3: Delete data files if cleanup level is kAll.
+ if (level == CleanupLevel::kAll && !manifests_to_delete.empty()) {
+ auto data_files_result =
+ FindDataFilesToDelete(metadata, manifests_to_delete, retained_manifest_paths);
+ if (data_files_result.has_value()) {
+ for (const auto& path : data_files_result.value()) {
+ DeleteFile(path);
+ }
+ }
+ }
+
+ // Phase 4: Delete orphaned manifest files.
+ for (const auto& path : manifests_to_delete) {
+ DeleteFile(path);
+ }
+
+ // Phase 5: Delete manifest lists from expired snapshots.
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) continue;
+ const auto& snapshot = snapshot_result.value();
+ if (!snapshot->manifest_list.empty()) {
+ DeleteFile(snapshot->manifest_list);
+ }
+ }
+
+ // Phase 6: Delete expired statistics files using path-based set difference.
+ for (const auto& path : ExpiredStatisticsFilePaths(metadata, expired_snapshot_ids)) {
+ DeleteFile(path);
+ }
+
+ return {};
+ }
+
+ private:
+ /// Cache of manifest path -> ManifestFile, populated during Phase 1 to avoid
+ /// re-reading manifest lists in FindDataFilesToDelete.
+ std::unordered_map<std::string, ManifestFile> manifest_cache_;
+
+ /// \brief Collect manifest paths for a snapshot into manifest_paths.
+ ///
+ /// Best-effort: if the snapshot or its manifest list cannot be read, the error
Review Comment:
[P0] Retained-manifest read failures can delete live manifests. `ReadManifestsForSnapshot` silently returns when a retained snapshot's manifest list cannot be read, but phase 2 still computes `manifests_to_delete` from that incomplete retained set and phase 4 deletes those manifests. That means a transient read failure while enumerating the current table state can delete manifest files that are still referenced by live snapshots. Java's `ReachableFileCleanup` aborts cleanup on this path (`throwFailureWhenFinished`) specifically to avoid deleting live metadata, so we should fail or skip manifest deletion entirely when retained manifests cannot be enumerated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r2996574718
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.insert(entry.data_file->file_path);
+ }
+ }
+ goto next_manifest; // Found and processed this manifest, move to next
Review Comment:
Removed. The nested loop + goto was eliminated by introducing `manifest_cache_` (populated in Phase 1) and a `MakeManifestReader()` helper. `FindDataFilesToDelete()` now iterates over the cache directly with simple map lookups.
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
Review Comment:
Changed `FindDataFilesToDelete` to return `Result<std::unordered_set<std::string>>` instead of using a mutable out-parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r3035741481
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,250 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Result<std::shared_ptr<ManifestReader>> ExpireSnapshots::MakeManifestReader(
Review Comment:
Done — MakeManifestReader is now a free function in the anonymous namespace.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r2996573634
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.insert(entry.data_file->file_path);
+ }
+ }
+ goto next_manifest; // Found and processed this manifest, move to next
+ }
+ }
+ next_manifest:;
+ }
+
+ if (data_files_to_delete.empty()) {
+ return {};
+ }
+
+ // Step 2: Remove any files that are still referenced by retained manifests.
+ // This ensures we don't delete files that are shared across manifests.
+ for (const auto& manifest_path : retained_manifests) {
+ if (data_files_to_delete.empty()) break;
+
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r3104002403
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -37,6 +43,264 @@
namespace iceberg {
+namespace {
+
+Result<std::shared_ptr<ManifestReader>> MakeManifestReader(
+ const ManifestFile& manifest, const std::shared_ptr<FileIO>& file_io,
+ const TableMetadata& metadata) {
+ ICEBERG_ASSIGN_OR_RAISE(auto schema, metadata.Schema());
+ ICEBERG_ASSIGN_OR_RAISE(auto spec,
+ metadata.PartitionSpecById(manifest.partition_spec_id));
+ return ManifestReader::Make(manifest, file_io, std::move(schema), std::move(spec));
+}
+
+/// \brief Abstract strategy for cleaning up files after snapshot expiration.
+///
+/// Mirrors Java's FileCleanupStrategy: provides shared delete utilities while
+/// allowing different cleanup algorithms (ReachableFileCleanup, IncrementalFileCleanup).
+class FileCleanupStrategy {
+ public:
+ FileCleanupStrategy(std::shared_ptr<FileIO> file_io,
+ std::function<void(const std::string&)> delete_func)
+ : file_io_(std::move(file_io)), delete_func_(std::move(delete_func)) {}
+
+ virtual ~FileCleanupStrategy() = default;
+
+ /// \brief Clean up files that are only reachable by expired snapshots.
+ ///
+ /// \param metadata Table metadata before expiration (contains all snapshots).
+ /// \param expired_snapshot_ids Snapshot IDs that were expired during this operation.
+ /// \param level Controls which types of files are eligible for deletion.
+ virtual Status CleanFiles(const TableMetadata& metadata,
+ const std::unordered_set<int64_t>& expired_snapshot_ids,
+ CleanupLevel level) = 0;
+
+ protected:
+ /// \brief Delete a file, suppressing errors (best-effort).
+ ///
+ /// Uses the custom delete function if set, otherwise FileIO::DeleteFile.
+ /// Matches Java's suppressFailureWhenFinished behavior.
+ void DeleteFile(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ std::ignore = file_io_->DeleteFile(path);
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+ }
+
+ /// \brief Returns paths of statistics files referenced only by expired snapshots.
+ ///
+ /// Uses path-based set difference (matching Java's expiredStatisticsFilesLocations):
+ /// if the same file path is shared across snapshots, it is only deleted when
+ /// no retained snapshot references it.
+ std::unordered_set<std::string> ExpiredStatisticsFilePaths(
+ const TableMetadata& metadata, const std::unordered_set<int64_t>& expired_ids) {
+ std::unordered_set<std::string> retained_paths;
+ for (const auto& stat : metadata.statistics) {
+ if (stat && !expired_ids.contains(stat->snapshot_id)) {
+ retained_paths.insert(stat->path);
+ }
+ }
+ for (const auto& part_stat : metadata.partition_statistics) {
+ if (part_stat && !expired_ids.contains(part_stat->snapshot_id)) {
+ retained_paths.insert(part_stat->path);
+ }
+ }
+
+ std::unordered_set<std::string> expired_paths;
+ for (const auto& stat : metadata.statistics) {
+ if (stat && expired_ids.contains(stat->snapshot_id) &&
+ !retained_paths.contains(stat->path)) {
+ expired_paths.insert(stat->path);
+ }
+ }
+ for (const auto& part_stat : metadata.partition_statistics) {
+ if (part_stat && expired_ids.contains(part_stat->snapshot_id) &&
+ !retained_paths.contains(part_stat->path)) {
+ expired_paths.insert(part_stat->path);
+ }
+ }
+ return expired_paths;
+ }
+
+ std::shared_ptr<FileIO> file_io_;
+ std::function<void(const std::string&)> delete_func_;
+};
+
+/// \brief File cleanup strategy that determines safe deletions via full reachability.
+///
+/// Mirrors Java's ReachableFileCleanup: collects manifests from all expired and
+/// retained snapshots, prunes candidates still referenced by retained snapshots,
+/// then deletes orphaned manifests, data files, manifest lists, and statistics files.
+///
+/// TODO(shangxinli): Add multi-threaded manifest reading and file deletion support.
+class ReachableFileCleanup : public FileCleanupStrategy {
+ public:
+ using FileCleanupStrategy::FileCleanupStrategy;
+
+ Status CleanFiles(const TableMetadata& metadata,
+ const std::unordered_set<int64_t>& expired_snapshot_ids,
+ CleanupLevel level) override {
+ std::unordered_set<int64_t> retained_snapshot_ids;
+ for (const auto& snapshot : metadata.snapshots) {
+ if (snapshot && !expired_snapshot_ids.contains(snapshot->snapshot_id)) {
+ retained_snapshot_ids.insert(snapshot->snapshot_id);
+ }
+ }
+
+ // Phase 1: Collect manifest paths from expired and retained snapshots.
+ // The manifest_cache_ is populated here to avoid O(M*S) repeated I/O in
+ // FindDataFilesToDelete.
+ std::unordered_set<std::string> expired_manifest_paths;
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ ReadManifestsForSnapshot(metadata, snapshot_id, expired_manifest_paths);
+ }
+ std::unordered_set<std::string> retained_manifest_paths;
+ for (int64_t snapshot_id : retained_snapshot_ids) {
+ ReadManifestsForSnapshot(metadata, snapshot_id, retained_manifest_paths);
+ }
+
+ // Phase 2: Prune manifests still referenced by retained snapshots.
+ std::unordered_set<std::string> manifests_to_delete;
+ for (const auto& path : expired_manifest_paths) {
+ if (!retained_manifest_paths.contains(path)) {
+ manifests_to_delete.insert(path);
+ }
+ }
+
+ // Phase 3: Delete data files if cleanup level is kAll.
+ if (level == CleanupLevel::kAll && !manifests_to_delete.empty()) {
+ auto data_files_result =
+ FindDataFilesToDelete(metadata, manifests_to_delete, retained_manifest_paths);
+ if (data_files_result.has_value()) {
+ for (const auto& path : data_files_result.value()) {
+ DeleteFile(path);
+ }
+ }
+ }
+
+ // Phase 4: Delete orphaned manifest files.
+ for (const auto& path : manifests_to_delete) {
+ DeleteFile(path);
+ }
+
+ // Phase 5: Delete manifest lists from expired snapshots.
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) continue;
+ const auto& snapshot = snapshot_result.value();
+ if (!snapshot->manifest_list.empty()) {
+ DeleteFile(snapshot->manifest_list);
+ }
+ }
+
+ // Phase 6: Delete expired statistics files using path-based set difference.
+ for (const auto& path : ExpiredStatisticsFilePaths(metadata, expired_snapshot_ids)) {
+ DeleteFile(path);
+ }
+
+ return {};
+ }
+
+ private:
+ /// Cache of manifest path -> ManifestFile, populated during Phase 1 to avoid
+ /// re-reading manifest lists in FindDataFilesToDelete.
+ std::unordered_map<std::string, ManifestFile> manifest_cache_;
+
+ /// \brief Collect manifest paths for a snapshot into manifest_paths.
+ ///
+ /// Best-effort: if the snapshot or its manifest list cannot be read, the error
Review Comment:
Fixed. ReadManifestsForSnapshot now returns bool. In CleanFiles, I track retained_manifests_complete; if any retained snapshot's manifest list can't be read the entire manifest/data-file deletion block (phases 2–4) is skipped, matching Java's throwFailureWhenFinished guard. Manifest-list deletion (phase 5) is unaffected since it depends only on expired snapshots.
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -37,6 +43,264 @@
namespace iceberg {
+namespace {
+
+Result<std::shared_ptr<ManifestReader>> MakeManifestReader(
+ const ManifestFile& manifest, const std::shared_ptr<FileIO>& file_io,
+ const TableMetadata& metadata) {
+ ICEBERG_ASSIGN_OR_RAISE(auto schema, metadata.Schema());
+ ICEBERG_ASSIGN_OR_RAISE(auto spec,
+ metadata.PartitionSpecById(manifest.partition_spec_id));
+ return ManifestReader::Make(manifest, file_io, std::move(schema), std::move(spec));
+}
+
+/// \brief Abstract strategy for cleaning up files after snapshot expiration.
+///
+/// Mirrors Java's FileCleanupStrategy: provides shared delete utilities while
+/// allowing different cleanup algorithms (ReachableFileCleanup, IncrementalFileCleanup).
+class FileCleanupStrategy {
+ public:
+ FileCleanupStrategy(std::shared_ptr<FileIO> file_io,
+ std::function<void(const std::string&)> delete_func)
+ : file_io_(std::move(file_io)), delete_func_(std::move(delete_func)) {}
+
+ virtual ~FileCleanupStrategy() = default;
+
+ /// \brief Clean up files that are only reachable by expired snapshots.
+ ///
+ /// \param metadata Table metadata before expiration (contains all snapshots).
+ /// \param expired_snapshot_ids Snapshot IDs that were expired during this operation.
+ /// \param level Controls which types of files are eligible for deletion.
+ virtual Status CleanFiles(const TableMetadata& metadata,
+ const std::unordered_set<int64_t>& expired_snapshot_ids,
+ CleanupLevel level) = 0;
+
+ protected:
+ /// \brief Delete a file, suppressing errors (best-effort).
+ ///
+ /// Uses the custom delete function if set, otherwise FileIO::DeleteFile.
+ /// Matches Java's suppressFailureWhenFinished behavior.
+ void DeleteFile(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ std::ignore = file_io_->DeleteFile(path);
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+ }
+
+ /// \brief Returns paths of statistics files referenced only by expired snapshots.
+ ///
+ /// Uses path-based set difference (matching Java's expiredStatisticsFilesLocations):
+ /// if the same file path is shared across snapshots, it is only deleted when
+ /// no retained snapshot references it.
+ std::unordered_set<std::string> ExpiredStatisticsFilePaths(
+ const TableMetadata& metadata, const std::unordered_set<int64_t>& expired_ids) {
+ std::unordered_set<std::string> retained_paths;
+ for (const auto& stat : metadata.statistics) {
+ if (stat && !expired_ids.contains(stat->snapshot_id)) {
+ retained_paths.insert(stat->path);
+ }
+ }
+ for (const auto& part_stat : metadata.partition_statistics) {
+ if (part_stat && !expired_ids.contains(part_stat->snapshot_id)) {
+ retained_paths.insert(part_stat->path);
+ }
+ }
+
+ std::unordered_set<std::string> expired_paths;
+ for (const auto& stat : metadata.statistics) {
+ if (stat && expired_ids.contains(stat->snapshot_id) &&
+ !retained_paths.contains(stat->path)) {
+ expired_paths.insert(stat->path);
+ }
+ }
+ for (const auto& part_stat : metadata.partition_statistics) {
+ if (part_stat && expired_ids.contains(part_stat->snapshot_id) &&
+ !retained_paths.contains(part_stat->path)) {
+ expired_paths.insert(part_stat->path);
+ }
+ }
+ return expired_paths;
+ }
+
+ std::shared_ptr<FileIO> file_io_;
+ std::function<void(const std::string&)> delete_func_;
+};
+
+/// \brief File cleanup strategy that determines safe deletions via full reachability.
+///
+/// Mirrors Java's ReachableFileCleanup: collects manifests from all expired and
+/// retained snapshots, prunes candidates still referenced by retained snapshots,
+/// then deletes orphaned manifests, data files, manifest lists, and statistics files.
+///
+/// TODO(shangxinli): Add multi-threaded manifest reading and file deletion support.
+class ReachableFileCleanup : public FileCleanupStrategy {
+ public:
+ using FileCleanupStrategy::FileCleanupStrategy;
+
+ Status CleanFiles(const TableMetadata& metadata,
+ const std::unordered_set<int64_t>& expired_snapshot_ids,
+ CleanupLevel level) override {
+ std::unordered_set<int64_t> retained_snapshot_ids;
+ for (const auto& snapshot : metadata.snapshots) {
+ if (snapshot && !expired_snapshot_ids.contains(snapshot->snapshot_id)) {
+ retained_snapshot_ids.insert(snapshot->snapshot_id);
+ }
+ }
+
+ // Phase 1: Collect manifest paths from expired and retained snapshots.
+ // The manifest_cache_ is populated here to avoid O(M*S) repeated I/O in
+ // FindDataFilesToDelete.
+ std::unordered_set<std::string> expired_manifest_paths;
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ ReadManifestsForSnapshot(metadata, snapshot_id, expired_manifest_paths);
+ }
+ std::unordered_set<std::string> retained_manifest_paths;
+ for (int64_t snapshot_id : retained_snapshot_ids) {
+ ReadManifestsForSnapshot(metadata, snapshot_id, retained_manifest_paths);
+ }
+
+ // Phase 2: Prune manifests still referenced by retained snapshots.
+ std::unordered_set<std::string> manifests_to_delete;
+ for (const auto& path : expired_manifest_paths) {
+ if (!retained_manifest_paths.contains(path)) {
+ manifests_to_delete.insert(path);
+ }
+ }
+
+ // Phase 3: Delete data files if cleanup level is kAll.
+ if (level == CleanupLevel::kAll && !manifests_to_delete.empty()) {
+ auto data_files_result =
+ FindDataFilesToDelete(metadata, manifests_to_delete, retained_manifest_paths);
+ if (data_files_result.has_value()) {
+ for (const auto& path : data_files_result.value()) {
+ DeleteFile(path);
+ }
+ }
+ }
+
+ // Phase 4: Delete orphaned manifest files.
+ for (const auto& path : manifests_to_delete) {
+ DeleteFile(path);
+ }
+
+ // Phase 5: Delete manifest lists from expired snapshots.
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) continue;
+ const auto& snapshot = snapshot_result.value();
+ if (!snapshot->manifest_list.empty()) {
+ DeleteFile(snapshot->manifest_list);
+ }
+ }
+
+ // Phase 6: Delete expired statistics files using path-based set difference.
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r3035737837
##########
src/iceberg/update/expire_snapshots.h:
##########
@@ -22,10 +22,14 @@
#include <cstdint>
#include <functional>
#include <memory>
+#include <string>
+#include <unordered_map>
#include <unordered_set>
#include <vector>
#include "iceberg/iceberg_export.h"
+#include "iceberg/manifest/manifest_list.h"
Review Comment:
Done — removed manifest_list.h and manifest_reader.h from the header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r2996571613
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.insert(entry.data_file->file_path);
+ }
+ }
+ goto next_manifest; // Found and processed this manifest, move to next
+ }
+ }
+ next_manifest:;
+ }
+
+ if (data_files_to_delete.empty()) {
+ return {};
+ }
+
+ // Step 2: Remove any files that are still referenced by retained manifests.
+ // This ensures we don't delete files that are shared across manifests.
+ for (const auto& manifest_path : retained_manifests) {
+ if (data_files_to_delete.empty()) break;
+
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
Review Comment:
Fixed — changed to `LiveEntries()` here as well, consistent with Step 1.
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.insert(entry.data_file->file_path);
+ }
+ }
+ goto next_manifest; // Found and processed this manifest, move to next
+ }
+ }
+ next_manifest:;
+ }
+
+ if (data_files_to_delete.empty()) {
+ return {};
+ }
+
+ // Step 2: Remove any files that are still referenced by retained manifests.
+ // This ensures we don't delete files that are shared across manifests.
+ for (const auto& manifest_path : retained_manifests) {
+ if (data_files_to_delete.empty()) break;
+
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.erase(entry.data_file->file_path);
+ }
+ }
+ goto next_retained;
+ }
+ }
+ next_retained:;
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::CleanExpiredFiles(
+ const std::vector<int64_t>& expired_snapshot_ids) {
+ const TableMetadata& metadata = base();
+
+ // Build expired and retained snapshot ID sets.
+ // The retained set includes ALL snapshots referenced by any branch or tag,
+ // since Apply() already computed retention across all refs.
+ std::unordered_set<int64_t> expired_id_set(expired_snapshot_ids.begin(),
+ expired_snapshot_ids.end());
+ std::unordered_set<int64_t> retained_snapshot_ids;
+ for (const auto& snapshot : metadata.snapshots) {
+ if (snapshot && !expired_id_set.contains(snapshot->snapshot_id)) {
+ retained_snapshot_ids.insert(snapshot->snapshot_id);
+ }
+ }
+
+ // Phase 1: Collect manifest paths from expired and retained snapshots.
+ // TODO(shangxinli): Parallelize manifest collection with a thread pool.
+ std::unordered_set<std::string> expired_manifest_paths;
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ std::ignore = ReadManifestsForSnapshot(snapshot_id, expired_manifest_paths);
+ }
+
+ std::unordered_set<std::string> retained_manifest_paths;
+ for (int64_t snapshot_id : retained_snapshot_ids) {
+ std::ignore = ReadManifestsForSnapshot(snapshot_id, retained_manifest_paths);
+ }
+
+ // Phase 2: Prune manifests still referenced by retained snapshots.
+ // Only manifests exclusively in expired snapshots should be deleted.
+ std::unordered_set<std::string> manifests_to_delete;
+ for (const auto& path : expired_manifest_paths) {
+ if (!retained_manifest_paths.contains(path)) {
+ manifests_to_delete.insert(path);
+ }
+ }
+
+ // Phase 3: If cleanup level is kAll, find data files to delete.
+ // Only read entries from manifests being deleted (not all expired manifests),
+ // then subtract any files still reachable from retained manifests.
+ if (cleanup_level_ == CleanupLevel::kAll && !manifests_to_delete.empty()) {
+ std::unordered_set<std::string> data_files_to_delete;
+ std::ignore = FindDataFilesToDelete(manifests_to_delete, retained_manifest_paths,
+ data_files_to_delete);
+
+ // TODO(shangxinli): Parallelize file deletion with a thread pool.
+ for (const auto& path : data_files_to_delete) {
+ DeleteFilePath(path);
+ }
+ }
+
+ // Phase 4: Delete orphaned manifest files.
+ for (const auto& path : manifests_to_delete) {
+ DeleteFilePath(path);
+ }
+
+ // Phase 5: Delete manifest lists from expired snapshots.
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) continue;
+ auto& snapshot = snapshot_result.value();
+ if (!snapshot->manifest_list.empty()) {
+ DeleteFilePath(snapshot->manifest_list);
+ }
+ }
+
+ // Phase 6: Delete expired statistics files.
+ // Use set difference between before and after states (matching Java behavior).
+ // Since Finalize runs before table_ is updated, "after" is base() minus expired.
+ std::unordered_set<int64_t> retained_stats_snapshots(retained_snapshot_ids);
+ for (const auto& stat_file : metadata.statistics) {
+ if (stat_file && !retained_stats_snapshots.contains(stat_file->snapshot_id)) {
Review Comment:
Good point. Changed to path-based set difference: collect paths from retained snapshots first, then only delete expired statistics files whose paths are not in the retained set. This prevents erroneous deletion when a statistics file path is shared across snapshots
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r2996567900
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
Review Comment:
Good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r3035800762
##########
src/iceberg/update/expire_snapshots.h:
##########
@@ -169,6 +224,14 @@ class ICEBERG_EXPORT ExpireSnapshots : public PendingUpdate {
enum CleanupLevel cleanup_level_ { CleanupLevel::kAll };
bool clean_expired_metadata_{false};
bool specified_snapshot_id_{false};
+
+ /// Cached result from Apply(), used during Finalize() for file cleanup
+ std::optional<ApplyResult> apply_result_;
+
+ /// Cache of manifest path -> ManifestFile, built during ReadManifestsForSnapshot
+ /// to avoid O(M*S) repeated I/O from re-reading manifest lists in
+ /// FindDataFilesToDelete.
+ std::unordered_map<std::string, ManifestFile> manifest_cache_;
Review Comment:
Done — manifest_cache_ is now a private member of ReachableFileCleanup in expire_snapshots.cc. It no longer appears in the header at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#issuecomment-4208156061
@wgtmac do you have time to have another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r2996569153
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
Review Comment:
changed to `LiveEntries()` to match Java's `ManifestFiles.readPaths` which delegates to `liveEntries()` (only ADDED and EXISTING). Using `Entries()` would include DELETED entries and cause storage leaks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r2996570443
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,247 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ }
+
+ return {};
+}
+
+Status ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests,
+ std::unordered_set<std::string>& data_files_to_delete) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ // Step 1: Collect all file paths from manifests being deleted
+ for (const auto& manifest_path : manifests_to_delete) {
+ // Find the ManifestFile for this path by scanning expired snapshots
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) continue;
+
+ for (const auto& manifest : manifests_result.value()) {
+ if (manifest.manifest_path != manifest_path) continue;
+
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) continue;
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) continue;
+
+ auto reader_result = ManifestReader::Make(
+ manifest, file_io, schema_result.value(), spec_result.value());
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->Entries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.insert(entry.data_file->file_path);
+ }
+ }
+ goto next_manifest; // Found and processed this manifest, move to next
+ }
+ }
+ next_manifest:;
+ }
+
+ if (data_files_to_delete.empty()) {
+ return {};
+ }
+
+ // Step 2: Remove any files that are still referenced by retained manifests.
+ // This ensures we don't delete files that are shared across manifests.
+ for (const auto& manifest_path : retained_manifests) {
+ if (data_files_to_delete.empty()) break;
+
+ for (const auto& snapshot : metadata.snapshots) {
+ if (!snapshot) continue;
+ SnapshotCache snapshot_cache(snapshot.get());
Review Comment:
Same as #1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#issuecomment-4163626854
I'm on the way :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add file cleanup for expire snapshots [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #592:
URL: https://github.com/apache/iceberg-cpp/pull/592#discussion_r3020581589
##########
src/iceberg/update/expire_snapshots.h:
##########
@@ -22,10 +22,14 @@
#include <cstdint>
#include <functional>
#include <memory>
+#include <string>
+#include <unordered_map>
#include <unordered_set>
#include <vector>
#include "iceberg/iceberg_export.h"
+#include "iceberg/manifest/manifest_list.h"
Review Comment:
Can we remove these includes by using forward declaration? `iceberg/type_fwd.h` has already been included.
##########
src/iceberg/update/expire_snapshots.h:
##########
@@ -169,6 +224,14 @@ class ICEBERG_EXPORT ExpireSnapshots : public PendingUpdate {
enum CleanupLevel cleanup_level_ { CleanupLevel::kAll };
bool clean_expired_metadata_{false};
bool specified_snapshot_id_{false};
+
+ /// Cached result from Apply(), used during Finalize() for file cleanup
+ std::optional<ApplyResult> apply_result_;
+
+ /// Cache of manifest path -> ManifestFile, built during ReadManifestsForSnapshot
+ /// to avoid O(M*S) repeated I/O from re-reading manifest lists in
+ /// FindDataFilesToDelete.
+ std::unordered_map<std::string, ManifestFile> manifest_cache_;
Review Comment:
This is the internal and ephemeral state used when cleaning up files. Can we remove it from here and move to the source file (or concrete FileCleanupStrategy class)?
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,250 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
Review Comment:
```suggestion
std::ignore = ctx_->table->io()->DeleteFile(path);
```
##########
src/iceberg/update/expire_snapshots.h:
##########
@@ -159,6 +173,47 @@ class ICEBERG_EXPORT ExpireSnapshots : public PendingUpdate {
Result<std::unordered_set<int64_t>> UnreferencedSnapshotIdsToRetain(
const SnapshotToRef& refs) const;
+ /// \brief Clean up files no longer referenced after snapshot expiration.
+ ///
+ /// Implements the "reachable file cleanup" strategy from Java's ReachableFileCleanup:
+ /// 1. Collect manifests from expired and retained snapshots
+ /// 2. Prune manifests still referenced by retained snapshots
+ /// 3. Find data files only in manifests being deleted (if kAll)
+ /// 4. Remove data files still reachable from retained manifests
+ /// 5. Delete orphaned manifests, manifest lists, and statistics files
+ ///
+ /// All deletions are best-effort: failures are suppressed to avoid blocking
+ /// metadata updates (matching Java's suppressFailureWhenFinished behavior).
+ ///
+ /// Branch/tag awareness: retained_snapshot_ids includes all snapshots referenced
+ /// by any branch or tag, as computed by Apply(). This prevents deleting files
+ /// that are still reachable from any ref.
+ ///
+ /// TODO(shangxinli): Add multi-threaded file deletion support.
+ /// TODO(shangxinli): Add IncrementalFileCleanup strategy for linear ancestry.
+ Status CleanExpiredFiles(const std::vector<int64_t>& expired_snapshot_ids);
+
+ /// \brief Read manifest paths from a single snapshot.
+ /// Best-effort: returns OK even if the snapshot or its manifests can't be read.
+ Status ReadManifestsForSnapshot(int64_t snapshot_id,
+ std::unordered_set<std::string>& manifest_paths);
+
+ /// \brief Find data files to delete by reading live entries from manifests being
+ /// deleted, then subtracting files still reachable from retained manifests.
+ /// If a retained manifest cannot be read, returns an empty set to prevent
+ /// accidental data loss.
+ Result<std::unordered_set<std::string>> FindDataFilesToDelete(
Review Comment:
This function seems to be related to ReachableFileCleanup only.
##########
src/iceberg/update/expire_snapshots.h:
##########
@@ -159,6 +173,47 @@ class ICEBERG_EXPORT ExpireSnapshots : public PendingUpdate {
Result<std::unordered_set<int64_t>> UnreferencedSnapshotIdsToRetain(
const SnapshotToRef& refs) const;
+ /// \brief Clean up files no longer referenced after snapshot expiration.
+ ///
+ /// Implements the "reachable file cleanup" strategy from Java's ReachableFileCleanup:
+ /// 1. Collect manifests from expired and retained snapshots
+ /// 2. Prune manifests still referenced by retained snapshots
+ /// 3. Find data files only in manifests being deleted (if kAll)
+ /// 4. Remove data files still reachable from retained manifests
+ /// 5. Delete orphaned manifests, manifest lists, and statistics files
+ ///
+ /// All deletions are best-effort: failures are suppressed to avoid blocking
+ /// metadata updates (matching Java's suppressFailureWhenFinished behavior).
+ ///
+ /// Branch/tag awareness: retained_snapshot_ids includes all snapshots referenced
+ /// by any branch or tag, as computed by Apply(). This prevents deleting files
+ /// that are still reachable from any ref.
+ ///
+ /// TODO(shangxinli): Add multi-threaded file deletion support.
+ /// TODO(shangxinli): Add IncrementalFileCleanup strategy for linear ancestry.
+ Status CleanExpiredFiles(const std::vector<int64_t>& expired_snapshot_ids);
+
+ /// \brief Read manifest paths from a single snapshot.
+ /// Best-effort: returns OK even if the snapshot or its manifests can't be read.
+ Status ReadManifestsForSnapshot(int64_t snapshot_id,
+ std::unordered_set<std::string>& manifest_paths);
+
+ /// \brief Find data files to delete by reading live entries from manifests being
+ /// deleted, then subtracting files still reachable from retained manifests.
+ /// If a retained manifest cannot be read, returns an empty set to prevent
+ /// accidental data loss.
+ Result<std::unordered_set<std::string>> FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests);
+
+ /// \brief Create a ManifestReader for the given ManifestFile.
+ Result<std::shared_ptr<ManifestReader>> MakeManifestReader(
Review Comment:
This function doesn't justify to be a member function.
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,250 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Result<std::shared_ptr<ManifestReader>> ExpireSnapshots::MakeManifestReader(
Review Comment:
Make it a local function in the anonymous namespace?
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,250 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Result<std::shared_ptr<ManifestReader>> ExpireSnapshots::MakeManifestReader(
+ const ManifestFile& manifest, const std::shared_ptr<FileIO>& file_io) {
+ const TableMetadata& metadata = base();
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) return std::unexpected<Error>(schema_result.error());
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) return std::unexpected<Error>(spec_result.error());
+ return ManifestReader::Make(manifest, file_io, schema_result.value(),
+ spec_result.value());
+}
+
+Status ExpireSnapshots::ReadManifestsForSnapshot(
+ int64_t snapshot_id, std::unordered_set<std::string>& manifest_paths) {
+ const TableMetadata& metadata = base();
+ auto file_io = ctx_->table->io();
+
+ auto snapshot_result = metadata.SnapshotById(snapshot_id);
+ if (!snapshot_result.has_value()) {
+ return {};
+ }
+ auto& snapshot = snapshot_result.value();
+
+ SnapshotCache snapshot_cache(snapshot.get());
+ auto manifests_result = snapshot_cache.Manifests(file_io);
+ if (!manifests_result.has_value()) {
+ // Best-effort: skip this snapshot if we can't read its manifests
+ return {};
+ }
+
+ for (const auto& manifest : manifests_result.value()) {
+ manifest_paths.insert(manifest.manifest_path);
+ // Cache manifest metadata for later use in FindDataFilesToDelete,
+ // avoiding O(M*S) repeated I/O from re-reading manifest lists.
+ manifest_cache_.emplace(manifest.manifest_path, manifest);
+ }
+
+ return {};
+}
+
+Result<std::unordered_set<std::string>> ExpireSnapshots::FindDataFilesToDelete(
+ const std::unordered_set<std::string>& manifests_to_delete,
+ const std::unordered_set<std::string>& retained_manifests) {
+ auto file_io = ctx_->table->io();
+ std::unordered_set<std::string> data_files_to_delete;
+
+ // Step 1: Collect live file paths from manifests being deleted.
+ // Use LiveEntries() (ADDED/EXISTING only) to match Java's ManifestFiles.readPaths
+ // which delegates to liveEntries(). Using Entries() would include DELETED entries
+ // and could cause storage leaks.
+ for (const auto& [path, manifest] : manifest_cache_) {
+ if (!manifests_to_delete.contains(path)) continue;
+
+ auto reader_result = MakeManifestReader(manifest, file_io);
+ if (!reader_result.has_value()) continue;
+
+ auto entries_result = reader_result.value()->LiveEntries();
+ if (!entries_result.has_value()) continue;
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.insert(entry.data_file->file_path);
+ }
+ }
+ }
+
+ if (data_files_to_delete.empty()) {
+ return data_files_to_delete;
+ }
+
+ // Step 2: Remove any files that are still referenced by retained manifests.
+ // If reading a retained manifest fails, we must NOT delete its data files
+ // to avoid accidental data loss (matching Java's retry + throwFailureWhenFinished).
+ for (const auto& manifest_path : retained_manifests) {
+ if (data_files_to_delete.empty()) break;
+
+ auto it = manifest_cache_.find(manifest_path);
+ if (it == manifest_cache_.end()) continue;
+
+ auto reader_result = MakeManifestReader(it->second, file_io);
+ if (!reader_result.has_value()) {
+ // Cannot read a retained manifest — abort data file deletion to prevent
+ // accidental data loss. Java retries and throws on failure here.
+ return std::unordered_set<std::string>{};
+ }
+
+ auto entries_result = reader_result.value()->LiveEntries();
+ if (!entries_result.has_value()) {
+ return std::unordered_set<std::string>{};
+ }
+
+ for (const auto& entry : entries_result.value()) {
+ if (entry.data_file) {
+ data_files_to_delete.erase(entry.data_file->file_path);
+ }
+ }
+ }
+
+ return data_files_to_delete;
+}
+
+Status ExpireSnapshots::CleanExpiredFiles(
+ const std::vector<int64_t>& expired_snapshot_ids) {
+ const TableMetadata& metadata = base();
+
+ // Build expired and retained snapshot ID sets.
+ // The retained set includes ALL snapshots referenced by any branch or tag,
+ // since Apply() already computed retention across all refs.
+ std::unordered_set<int64_t> expired_id_set(expired_snapshot_ids.begin(),
+ expired_snapshot_ids.end());
+ std::unordered_set<int64_t> retained_snapshot_ids;
+ for (const auto& snapshot : metadata.snapshots) {
+ if (snapshot && !expired_id_set.contains(snapshot->snapshot_id)) {
+ retained_snapshot_ids.insert(snapshot->snapshot_id);
+ }
+ }
+
+ // Phase 1: Collect manifest paths from expired and retained snapshots.
+ // TODO(shangxinli): Parallelize manifest collection with a thread pool.
+ std::unordered_set<std::string> expired_manifest_paths;
+ for (int64_t snapshot_id : expired_snapshot_ids) {
+ std::ignore = ReadManifestsForSnapshot(snapshot_id, expired_manifest_paths);
Review Comment:
Why we can ignore the error? Isn't it safe to abort entire operation?
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,250 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
+ try {
+ if (delete_func_) {
+ delete_func_(path);
+ } else {
+ auto status = ctx_->table->io()->DeleteFile(path);
+ // Best-effort: ignore NotFound (file already deleted) and other errors.
+ // Java uses suppressFailureWhenFinished + onFailure logging.
+ std::ignore = status;
+ }
+ } catch (...) {
+ // Suppress all exceptions during file cleanup to match Java's
+ // suppressFailureWhenFinished behavior.
+ }
+}
+
+Result<std::shared_ptr<ManifestReader>> ExpireSnapshots::MakeManifestReader(
+ const ManifestFile& manifest, const std::shared_ptr<FileIO>& file_io) {
+ const TableMetadata& metadata = base();
+ auto schema_result = metadata.Schema();
+ if (!schema_result.has_value()) return std::unexpected<Error>(schema_result.error());
+ auto spec_result = metadata.PartitionSpecById(manifest.partition_spec_id);
+ if (!spec_result.has_value()) return std::unexpected<Error>(spec_result.error());
+ return ManifestReader::Make(manifest, file_io, schema_result.value(),
+ spec_result.value());
Review Comment:
```suggestion
ICEBERG_ASSIGN_OR_RAISE(auto schema, metadata.Schema());
ICEBERG_ASSIGN_OR_RAISE(auto spec, metadata.PartitionSpecById(manifest.partition_spec_id));
return ManifestReader::Make(manifest, file_io, std::move(schema), std::move(spec));
```
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,250 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
Review Comment:
Should we clear `apply_result_` if `CleanExpiredFiles` succeeds?
##########
src/iceberg/update/expire_snapshots.h:
##########
@@ -159,6 +173,47 @@ class ICEBERG_EXPORT ExpireSnapshots : public PendingUpdate {
Result<std::unordered_set<int64_t>> UnreferencedSnapshotIdsToRetain(
const SnapshotToRef& refs) const;
+ /// \brief Clean up files no longer referenced after snapshot expiration.
+ ///
+ /// Implements the "reachable file cleanup" strategy from Java's ReachableFileCleanup:
+ /// 1. Collect manifests from expired and retained snapshots
+ /// 2. Prune manifests still referenced by retained snapshots
+ /// 3. Find data files only in manifests being deleted (if kAll)
+ /// 4. Remove data files still reachable from retained manifests
+ /// 5. Delete orphaned manifests, manifest lists, and statistics files
+ ///
+ /// All deletions are best-effort: failures are suppressed to avoid blocking
+ /// metadata updates (matching Java's suppressFailureWhenFinished behavior).
+ ///
+ /// Branch/tag awareness: retained_snapshot_ids includes all snapshots referenced
+ /// by any branch or tag, as computed by Apply(). This prevents deleting files
+ /// that are still reachable from any ref.
+ ///
+ /// TODO(shangxinli): Add multi-threaded file deletion support.
+ /// TODO(shangxinli): Add IncrementalFileCleanup strategy for linear ancestry.
+ Status CleanExpiredFiles(const std::vector<int64_t>& expired_snapshot_ids);
+
+ /// \brief Read manifest paths from a single snapshot.
+ /// Best-effort: returns OK even if the snapshot or its manifests can't be read.
+ Status ReadManifestsForSnapshot(int64_t snapshot_id,
Review Comment:
It seems worth adding a similar abstraction like the Java `FileCleanupStrategy` if we will add `IncrementalFileCleanup` or more strategies in the future. We can add them only in the expire_snapshots.cc for now as it is unlikely to be used elsewhere.
I'm also thinking if it is valuable to enable users to customize any strategy by registering their own implementations but that's a separate topic.
##########
src/iceberg/update/expire_snapshots.cc:
##########
@@ -285,7 +291,250 @@ Result<ExpireSnapshots::ApplyResult> ExpireSnapshots::Apply() {
});
}
+ // Cache the result for use during Finalize()
+ apply_result_ = result;
+
return result;
}
+Status ExpireSnapshots::Finalize(std::optional<Error> commit_error) {
+ if (commit_error.has_value()) {
+ return {};
+ }
+
+ if (cleanup_level_ == CleanupLevel::kNone) {
+ return {};
+ }
+
+ if (!apply_result_.has_value() || apply_result_->snapshot_ids_to_remove.empty()) {
+ return {};
+ }
+
+ // File cleanup is best-effort: log and continue on individual file deletion failures
+ // to avoid blocking metadata updates (matching Java behavior).
+ return CleanExpiredFiles(apply_result_->snapshot_ids_to_remove);
+}
+
+void ExpireSnapshots::DeleteFilePath(const std::string& path) {
Review Comment:
```suggestion
void ExpireSnapshots::DeleteFile(const std::string& path) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org