You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhuxiangyi (via GitHub)" <gi...@apache.org> on 2026/09/26 12:03:51 UTC

[PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

zhuxiangyi opened a new pull request, #965:
URL: https://github.com/apache/paimon-rust/pull/965

   ### Purpose
   
   Linked issue: #964 (part 1 of 4)
   
   Tables written through paimon-rust (Rust API, pypaimon native, C FFI) never expire snapshots, so
   their snapshot, manifest, and data files only ever grow. This PR adds snapshot expiration to the
   core crate and exposes it as `CALL sys.expire_snapshots`, following Java's `ExpireSnapshotsImpl`
   and the Flink/Spark `expire_snapshots` procedure.
   
   ### Brief change log
   
   - `Table::new_expire_snapshots()` / `ExpireSnapshots` (`table/expire_snapshots.rs`):
     - Chooses snapshots to expire with Java's rules:
       - never more than `snapshot.num-retained.max` snapshots are kept, and never fewer than
         `snapshot.num-retained.min`;
       - a snapshot expires only once the snapshot after it is older than the cut-off
         (`snapshot.time-retained`, or an explicit `older_than`);
       - snapshots that a consumer (`consumer/consumer-*`) still reads from are kept;
       - one run expires at most `snapshot.expire.limit` snapshots.
   
       Each rule can be overridden per call.
     - Expiring `[earliest, end)` deletes, in Java's order:
       1. data files deleted by the delta manifests of `(earliest, end]` that the closest earlier
          tag does not still read;
       2. changelog files of `[earliest, end)`;
       3. manifest lists, manifests and their extra files, index manifests, index files, statistics
          files, and row-id reassign plans that neither `end` nor a tag in the range references;
       4. the snapshot files, and finally the EARLIEST hint.
   - `table/snapshot_deletion.rs` (Java `FileDeletionBase` / `SnapshotDeletion`) holds the planning
     and deletion. Every read failure makes the plan delete less, never more:
     - an unreadable delta manifest skips that snapshot's data files;
     - a tag that cannot be read skips the files it might protect;
     - a skipping set that cannot be built skips manifest deletion;
     - if the end snapshot disappears concurrently, the run stops before deleting any manifest.
   - Unlike Java, the check "a file the same delta re-adds (a level upgrade) is not deleted" does not
     depend on the DELETE entry coming before the ADD entry.
   - New `CoreOptions`: `snapshot.num-retained.min` / `.max`, `snapshot.time-retained`,
     `snapshot.expire.limit`, with Java's defaults. Invalid values fail instead of silently falling
     back, since this path deletes files.
   - `SnapshotManager::try_get_snapshot` and `earliest_hint_exists`.
   - DataFusion `CALL sys.expire_snapshots(table, retain_max, retain_min, older_than, max_deletes,
     options)` returns `deleted_snapshots_count`, like Java. `older_than` accepts epoch milliseconds
     or a `yyyy-MM-dd HH:mm:ss[.SSS]` timestamp in the local time zone (Java's format). `options`
     are dynamic table options for the call.
   - Uses `table.snapshot_manager()`, so REST-catalog tables resolve their latest snapshot through the
     catalog, as in Java.
   
   Scope notes:
   - Like Java's procedure, which builds its `ExpireConfig` without changelog options, changelog
     files expire with their snapshots. `changelog.num-retained.*` / `changelog.time-retained`
     (changelog decoupling) need a changelog manager and are left for later.
   - `snapshot.clean-empty-directories` is not implemented yet.
   
   ### Tests
   
   - `table::expire_snapshots::tests` (14 tests). The central check is
     `assert_files_match_references`: after a run, the data, manifest, and index files on disk must
     be exactly the files some remaining snapshot or tag references. Live data files come from the
     scan planner, not from the deletion code. The scenarios:
     - an empty table;
     - an append-only table, where every data file is kept;
     - overwrite, where the replaced files are deleted;
     - a partitioned overwrite, which checks partition path resolution;
     - a tag keeps its data files and manifests and stays readable through `scan.tag-name`;
     - a consumer keeps the snapshot it reads, which stays readable through `scan.snapshot-id`;
     - each retention rule on its own: `num-retained.max`, `older_than` (successor rule),
       `max_deletes`, `retain_min`;
     - invalid retention arguments and options are rejected;
     - a level upgrade (DELETE and ADD of the same file in one delta) keeps the file;
     - rewritten files are deleted;
     - a dropped global index's files are deleted while an index still in use is kept;
     - a missing end snapshot keeps all manifests;
     - the EARLIEST hint is written even when nothing expires;
     - `previous_tag` and `find_skipping_tags`.
   - The tests catch the key mistakes: disabling tag protection or upgrade protection each makes a
     test fail.
   - `CoreOptions` defaults, overrides, and invalid values.
   - DataFusion `tests/procedures.rs`:
     - an end-to-end `CALL sys.expire_snapshots` test covering `retain_max`, `older_than`,
       `max_deletes`, and `options`, checked against `$snapshots` and the table data;
     - argument validation: an unknown argument, a non-integer, `retain_max` below `retain_min`, and
       a bad timestamp.
   - A unit test for `older_than` parsing.
   - `cargo test -p paimon --all-targets --features fulltext,vortex` and `cargo clippy --all-targets
     --workspace --features fulltext,vortex -- -D warnings` pass.
   
   ### API and Format
   
   - New public API: `Table::new_expire_snapshots()` / `ExpireSnapshots`; new `CoreOptions`
     accessors; new `sys.expire_snapshots` procedure.
   - No storage format change.
   
   ### Documentation
   
   `docs/src/sql.md` documents `expire_snapshots`: its arguments, the table options behind them, and
   what the procedure keeps.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "zhuxiangyi (via GitHub)" <gi...@apache.org>.
zhuxiangyi closed pull request #965: feat(table): expire snapshots and add sys.expire_snapshots
URL: https://github.com/apache/paimon-rust/pull/965


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "zhuxiangyi (via GitHub)" <gi...@apache.org>.
zhuxiangyi commented on PR #965:
URL: https://github.com/apache/paimon-rust/pull/965#issuecomment-5981520090

   Round 3 is addressed in 5e9cbad (index-path API) and 22d9e7e (DST parsing); details are in the inline replies. The branch is rebased onto main 8c3527b, and the tests were run on that merge rather than on the older base:
   - the full core suite passes;
   - the DataFusion procedure tests and the New York DST test pass;
   - clippy with `-D warnings` passes.
   
   After the rebase, round 2's f7ba2d3 is da7bfd9.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #965:
URL: https://github.com/apache/paimon-rust/pull/965#discussion_r4203215361


##########
crates/paimon/src/table/expire_snapshots.rs:
##########
@@ -0,0 +1,468 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Snapshot expiration.
+//!
+//! Reference: Java `ExpireSnapshotsImpl`, configured like the Flink and Spark
+//! `expire_snapshots` procedures (`ProcedureUtils.fillInSnapshotOptions`).
+//!
+//! Expiring snapshots `[earliest, end)` removes, in order:
+//! 1. data files deleted by the delta manifests of `(earliest, end]` that no
+//!    tag still reads;
+//! 2. changelog files added by `[earliest, end)`;
+//! 3. manifest lists, manifests, index manifests, index files, statistics and
+//!    reassign plans of `[earliest, end)` that neither `end` nor a tag in the
+//!    range still references;
+//! 4. the snapshot files themselves, then the EARLIEST hint moves to `end`.
+//!
+//! Snapshot files go last, so an interrupted run leaves every remaining
+//! snapshot readable and a later run finishes the job.
+
+use crate::spec::{CoreOptions, Snapshot};
+use crate::table::snapshot_deletion::{read_long_lived_changelogs, DataFileKey, SnapshotDeletion};
+use crate::table::{BranchManager, Table};
+use crate::{Error, Result};
+use futures::{stream, StreamExt, TryStreamExt};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+use std::time::{SystemTime, UNIX_EPOCH};
+
+const SNAPSHOT_READ_CONCURRENCY: usize = 16;
+
+/// Retention rules for one expiration run. Java `ExpireConfig` (snapshot part).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+struct ExpireConfig {
+    retain_max: i32,
+    retain_min: i32,
+    /// Snapshots whose successor was committed at or after this time are kept.
+    older_than_millis: i64,
+    max_deletes: i32,
+}
+
+/// Expire old snapshots of a table and delete the files only they reference.
+///
+/// Defaults come from the table options `snapshot.num-retained.min`,
+/// `snapshot.num-retained.max`, `snapshot.time-retained` and
+/// `snapshot.expire.limit`; each can be overridden per run, like the arguments
+/// of Java's `expire_snapshots` procedure.
+pub struct ExpireSnapshots<'a> {
+    table: &'a Table,
+    retain_max: Option<i32>,
+    retain_min: Option<i32>,
+    older_than_millis: Option<i64>,
+    max_deletes: Option<i32>,
+    current_time_millis: Option<i64>,
+}
+
+impl<'a> ExpireSnapshots<'a> {
+    pub(crate) fn new(table: &'a Table) -> Self {
+        Self {
+            table,
+            retain_max: None,
+            retain_min: None,
+            older_than_millis: None,
+            max_deletes: None,
+            current_time_millis: None,
+        }
+    }
+
+    /// Maximum number of completed snapshots to retain.
+    pub fn with_retain_max(&mut self, retain_max: i32) -> &mut Self {
+        self.retain_max = Some(retain_max);
+        self
+    }
+
+    /// Minimum number of completed snapshots to retain.
+    pub fn with_retain_min(&mut self, retain_min: i32) -> &mut Self {
+        self.retain_min = Some(retain_min);
+        self
+    }
+
+    /// Expire snapshots older than this epoch-millisecond timestamp instead of
+    /// `snapshot.time-retained`.
+    pub fn with_older_than_millis(&mut self, older_than_millis: i64) -> &mut Self {
+        self.older_than_millis = Some(older_than_millis);
+        self
+    }
+
+    /// Maximum number of snapshots to expire in this run.
+    pub fn with_max_deletes(&mut self, max_deletes: i32) -> &mut Self {
+        self.max_deletes = Some(max_deletes);
+        self
+    }
+
+    #[cfg(test)]
+    pub(crate) fn with_current_time_millis(&mut self, current_time_millis: i64) -> &mut Self {
+        self.current_time_millis = Some(current_time_millis);
+        self
+    }
+
+    fn config(&self) -> Result<ExpireConfig> {
+        let core = CoreOptions::new(self.table.schema().options());
+        let retain_max = match self.retain_max {
+            Some(value) => positive("retain_max", value)?,
+            None => core.snapshot_num_retained_max()?,
+        };
+        let retain_min = match self.retain_min {
+            Some(value) => positive("retain_min", value)?,
+            None => core.snapshot_num_retained_min()?,
+        };
+        if retain_max < retain_min {
+            return Err(Error::DataInvalid {
+                message: format!(
+                    "retainMax ({retain_max}) must not be less than retainMin ({retain_min})."
+                ),
+                source: None,
+            });
+        }
+        let max_deletes = match self.max_deletes {
+            Some(value) => positive("max_deletes", value)?,
+            None => core.snapshot_expire_limit()?,
+        };
+        let older_than_millis = match self.older_than_millis {
+            Some(value) => value,
+            None => {
+                let now = self.current_time_millis.unwrap_or_else(current_time_millis);
+                let retained = i64::try_from(core.snapshot_time_retained_ms()?).unwrap_or(i64::MAX);
+                now.saturating_sub(retained)
+            }
+        };
+        Ok(ExpireConfig {
+            retain_max,
+            retain_min,
+            older_than_millis,
+            max_deletes,
+        })
+    }
+
+    /// Expire snapshots and return how many snapshot files were removed.
+    pub async fn execute(&self) -> Result<usize> {
+        self.table.ensure_not_branch_reference_for_write()?;
+        let config = self.config()?;
+
+        let snapshot_manager = self.table.snapshot_manager();
+        let Some(latest) = snapshot_manager.get_latest_snapshot().await? else {
+            return Ok(0);
+        };
+        let latest = latest.id();
+        let Some(earliest) = snapshot_manager.earliest_snapshot_id().await? else {
+            return Ok(0);
+        };
+
+        // The oldest snapshot `snapshot.num-retained.max` lets us keep.
+        let min = (latest - i64::from(config.retain_max) + 1).max(earliest);
+        // `snapshot.num-retained.min` protects the newest snapshots.
+        let mut max_exclusive = latest - i64::from(config.retain_min) + 1;
+        // A snapshot a consumer still reads from cannot be deleted.
+        let consumers = self.table.consumer_manager().list_all().await?;
+        if let Some(min_next) = consumers.iter().map(|(_, next)| *next).min() {
+            max_exclusive = max_exclusive.min(min_next);
+        }
+        // `snapshot.expire.limit` bounds one run.
+        max_exclusive = max_exclusive.min(earliest.saturating_add(i64::from(config.max_deletes)));
+
+        for id in min..max_exclusive {
+            // A snapshot expires only once its successor has also outlived the
+            // retention time.
+            if let Some(next) = snapshot_manager.try_get_snapshot(id + 1).await? {
+                if config.older_than_millis <= next.time_millis() as i64 {
+                    return self.expire_until(earliest, id).await;
+                }
+            }
+        }
+        self.expire_until(earliest, max_exclusive).await
+    }
+
+    async fn expire_until(&self, earliest: i64, end_exclusive: i64) -> Result<usize> {
+        let snapshot_manager = self.table.snapshot_manager();
+        if end_exclusive <= earliest {
+            // Nothing expires; record the earliest snapshot so later runs find
+            // it without listing the directory.
+            if !snapshot_manager.earliest_hint_exists().await? {
+                snapshot_manager.write_earliest_hint(earliest).await?;
+            }
+            return Ok(0);
+        }
+
+        // Futures are built eagerly: a borrowing closure inside the stream
+        // would make callers' futures lose `Send`.
+        let reads = (earliest..=end_exclusive)
+            .map(|id| snapshot_manager.try_get_snapshot(id))
+            .collect::<Vec<_>>();
+        let snapshots_including_end = stream::iter(reads)
+            .buffered(SNAPSHOT_READ_CONCURRENCY)
+            .try_collect::<Vec<_>>()
+            .await?
+            .into_iter()
+            .flatten()
+            .collect::<Vec<_>>();
+        let Some(first) = snapshots_including_end.first() else {
+            return Ok(0);
+        };
+        let begin_inclusive = first.id();
+        let snapshots_excluding_end = snapshots_including_end
+            .iter()
+            .filter(|snapshot| snapshot.id() != end_exclusive)
+            .collect::<Vec<_>>();
+
+        let tagged = self.tagged_snapshots().await?;
+        let deletion = SnapshotDeletion::new(self.table)?;
+        // Branches and long-lived changelogs share files with this history.
+        // Read everything they reference before deleting anything; if that is
+        // impossible, the run stops with nothing changed.
+        let external = self.external_owners(&deletion).await?;
+        // Their index and changelog manifests too: nothing may be deleted
+        // before every owner outside main's history has been read.
+        let external_snapshots = external.snapshots.iter().collect::<Vec<_>>();
+        let external_skipping = deletion
+            .manifest_skipping_set(&external_snapshots, true)
+            .await?;
+
+        // Data files deleted by a snapshot are unused from that snapshot on,
+        // so the range is (begin, end].
+        let data_snapshots = snapshots_including_end
+            .iter()
+            .filter(|snapshot| snapshot.id() != begin_inclusive)
+            .collect::<Vec<_>>();
+        self.clean_data_files(&deletion, &data_snapshots, &tagged, &external.data_files)
+            .await;
+
+        let mut changelog_files = Vec::new();
+        for snapshot in &snapshots_excluding_end {
+            let still_read = snapshot
+                .changelog_manifest_list()
+                .is_some_and(|list| external.changelog_lists.contains(list));
+            if !still_read {
+                changelog_files.extend(deletion.changelog_files(snapshot).await);
+            }
+        }
+        deletion.delete_quietly(changelog_files).await;
+
+        let last = snapshots_including_end
+            .last()
+            .expect("checked non-empty above");
+        if last.id() != end_exclusive {
+            // The end snapshot is gone, so nothing protects the manifests it
+            // would share; stop before deleting any of them.
+            return Ok(0);
+        }
+        let mut skipping_snapshots = find_skipping_tags(&tagged, begin_inclusive, end_exclusive);
+        skipping_snapshots.push(last);
+        match deletion
+            .manifest_skipping_set(&skipping_snapshots, false)
+            .await
+        {
+            Ok(mut skipping) => {
+                skipping.extend(external_skipping);
+                let mut manifest_files = Vec::new();
+                for snapshot in &snapshots_excluding_end {
+                    manifest_files.extend(
+                        deletion
+                            .unused_manifest_paths(snapshot, &mut skipping)
+                            .await?,
+                    );
+                }
+                deletion.delete_quietly(manifest_files).await;
+            }
+            Err(error) => log::info!(
+                "Skip cleaning manifest files because the skipping set cannot be built: {error}"
+            ),
+        }
+
+        for snapshot in &snapshots_excluding_end {
+            if let Err(error) = snapshot_manager.delete_snapshot(snapshot.id()).await {
+                log::warn!("Failed to delete snapshot {}: {error}", snapshot.id());
+            }
+        }
+        snapshot_manager.write_earliest_hint(end_exclusive).await?;
+        Ok(snapshots_excluding_end.len())
+    }
+
+    /// Delete the data files each snapshot's delta removes, keeping the ones
+    /// the closest earlier tag still reads. A file removed after a tag was taken
+    /// is live in that tag and in no later snapshot, so the closest earlier tag
+    /// is the only one that can still read it.
+    async fn clean_data_files(
+        &self,
+        deletion: &SnapshotDeletion,
+        snapshots: &[&Snapshot],
+        tagged: &[Snapshot],
+        external: &HashSet<DataFileKey>,
+    ) {
+        let plans = snapshots
+            .iter()
+            .map(|snapshot| deletion.plan_deleted_in_delta_manifest(snapshot))
+            .collect::<Vec<_>>();
+        let plans = stream::iter(plans)
+            .buffered(SNAPSHOT_READ_CONCURRENCY)
+            .collect::<Vec<_>>()
+            .await;
+
+        let mut tag_files: HashMap<i64, Option<HashSet<DataFileKey>>> = HashMap::new();
+        let mut paths = Vec::new();
+        for (snapshot, plan) in snapshots.iter().zip(&plans) {
+            let Some(tag) = previous_tag(tagged, snapshot.id()) else {
+                paths.extend(plan.paths_to_delete(&[external]));
+                continue;
+            };
+            if let Entry::Vacant(slot) = tag_files.entry(tag.id()) {
+                let files = match deletion.tag_data_files(tag).await {
+                    Ok(files) => Some(files),
+                    Err(error) => {
+                        log::info!(
+                            "Failed to read the data files of tag snapshot {}: {error}",
+                            tag.id()
+                        );
+                        None
+                    }
+                };
+                slot.insert(files);
+            }
+            match &tag_files[&tag.id()] {
+                Some(files) => paths.extend(plan.paths_to_delete(&[files, external])),
+                None => log::info!(
+                    "Skip cleaning data files of snapshot {} because tag snapshot {} cannot be read",
+                    snapshot.id(),
+                    tag.id()
+                ),
+            }
+        }
+        deletion.delete_quietly(paths).await;
+    }
+
+    /// Snapshots and tags of every branch, and long-lived changelogs of main
+    /// and every branch. Unlike main's own history, their files cannot be
+    /// reasoned about from the expired range, so everything they reference is
+    /// kept.
+    async fn external_owners(&self, deletion: &SnapshotDeletion) -> Result<ExternalOwners> {
+        let file_io = self.table.file_io();
+        let branch_manager = BranchManager::new(file_io.clone(), self.table.location().to_string());
+        let mut snapshots = read_long_lived_changelogs(file_io, self.table.location()).await?;
+        for branch in branch_manager.list_all().await? {
+            let snapshot_manager = self.table.snapshot_manager().with_branch(&branch);
+            for id in snapshot_manager.list_all_ids().await? {

Review Comment:
   [P1] Include each branch’s authoritative catalog snapshot before deleting files
   
   `list_all_ids`/`try_get_snapshot` enumerate filesystem snapshots only, whereas a REST-backed branch can read its latest snapshot exclusively through the catalog. I reproduced this with snapshot 1 retained by branch b1, no remaining main/branch tags, and a main overwrite creating snapshot 2. With branch schema present but its snapshot JSON absent from storage, public `copy_with_branch("b1")` reads row 1 successfully through REST before expiration. Keeping only one main snapshot then deletes that branch’s manifest list, and its next read fails with NotFound. The recorded GC requests contain only the main table snapshot endpoint, never the branch endpoint. This also fails on the automatic merge with current main; adding `snapshot_manager.get_latest_snapshot().await?` to the protected owner set makes the probe and all 30 original expiration tests pass. Please resolve the authoritative latest snapshot for every branch during the existing preflight, retaining the filesystem/tag/change
 log history as well and failing closed if catalog lookup fails.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "zhuxiangyi (via GitHub)" <gi...@apache.org>.
zhuxiangyi commented on PR #965:
URL: https://github.com/apache/paimon-rust/pull/965#issuecomment-5954969075

   Thanks for the detailed review and the probes. All three findings are fixed in e1d75d2, and the branch is rebased onto main (2d0dda1), with the `table/mod.rs` conflict resolved.
   
   **[P1] Branch references.** Before deleting anything, expiration now reads the snapshots and tags of every branch, and the long-lived changelogs of main and every branch. Everything these owners reference is kept:
   - the live data files they read;
   - their manifests, including changelog manifest lists and changelog manifests;
   - the changelog files of an expired snapshot whose changelog list an owner still names.
   
   If any of them cannot be read, the call fails before anything is changed, as you suggested.
   - `test_branch_keeps_the_files_it_shares_with_main` follows your reproduction. Snapshot 1 gets tag t1 and branch b1; t1 is deleted; main is overwritten and then expired. Main reads `[2]` and b1 still reads `[1]`.
   - `test_unreadable_branch_aborts_before_any_deletion` checks that if a branch's manifest cannot be read, the call fails and no snapshot, manifest, or data file is touched.
   
   **[P2] Persisted long-lived changelogs.** I chose to protect these owners rather than reject the run. `changelog/changelog-<id>` owners get the same treatment as branches above. `test_long_lived_changelog_keeps_its_files` uses an input-changelog table, persists `changelog-1`, commits snapshot 2, and expires. The changelog manifest list and every file it references remain. Applying the changelogs' own retention policy is still out of scope.
   
   **[P2] Legacy deletion-vector locations.** Before deleting index files, expiration now resolves DV entries with `resolve_legacy_deletion_vector_entries`, as reads do. `test_legacy_deletion_vector_location_is_expired` moves a DV to `table/index/`, checks that reads still find it, supersedes it, expires, and checks that the obsolete file is gone.
   
   Each new test fails when the protection it covers is disabled. The file-set invariant used by the existing tests now also counts branch and changelog owners. `cargo test -p paimon --all-targets --features fulltext,vortex` passes, and so do clippy with `-D warnings` and the DataFusion procedure tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #965:
URL: https://github.com/apache/paimon-rust/pull/965#issuecomment-5935442084

   Requirement fit: SUPPORTED. Implementation: FINDINGS.
   
   [P1] Protect every live branch before deleting shared files (`crates/paimon/src/table/expire_snapshots.rs:222-260`). The protection set is built only from the current branch's tags and retained end snapshot. A real writer/read probe creates snapshot 1, tag t1 and branch b1 from it, deletes the main t1 tag, overwrites main with snapshot 2, then expires main to one snapshot. Main remains readable with row 2, but b1, which read row 1 before expiration, now fails with NotFound for its original manifest list. Branch snapshot/tag metadata remains present, and its files are physically shared with main. Protect data and manifests referenced by every branch owner (not merely add branch tags to the closest-main-tag heuristic), or reject expiration when those references cannot be protected. This violates #964's explicit branch-reference safety rule; no Java-parity claim is needed. #966 further exposes this failure as an automatic commit side effect.
   
   [P2] Keep files referenced by persisted long-lived changelogs (`crates/paimon/src/table/expire_snapshots.rs:234-238`, `snapshot_deletion.rs:324-326`). Java's decoupled expiration writes changelog/changelog-<id> immediately before deleting snapshot-<id>, so an interrupted run legitimately leaves both owners. Changelog(Snapshot) retains its changelogManifestList, which Java incremental readers use. A real input-changelog probe persists that valid owner for snapshot 1, commits snapshot 2, and expires to one snapshot: changelog-1 remains but its previously existing changelog manifest list is deleted. Protect those persisted owners or reject this unsupported state before any mutation. Implementing their independent retention policy can remain out of scope; preventing corruption of an existing owner cannot be replaced by documenting that options are ignored.
   
   Validation at 695d3890687e477192631bdf59ab24e72ab109df: the original 23 core expiration tests pass; both additional writer/reference probes above fail. The same failures were independently reproduced through the real Python extension in #967. DataFusion procedure integration and read-failure/REST resolution were also checked. Current main conflicts in table/mod.rs, so integration needs a rebase and rerun before merge.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "zhuxiangyi (via GitHub)" <gi...@apache.org>.
zhuxiangyi commented on code in PR #965:
URL: https://github.com/apache/paimon-rust/pull/965#discussion_r4213547306


##########
crates/paimon/src/table/expire_snapshots.rs:
##########
@@ -0,0 +1,468 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Snapshot expiration.
+//!
+//! Reference: Java `ExpireSnapshotsImpl`, configured like the Flink and Spark
+//! `expire_snapshots` procedures (`ProcedureUtils.fillInSnapshotOptions`).
+//!
+//! Expiring snapshots `[earliest, end)` removes, in order:
+//! 1. data files deleted by the delta manifests of `(earliest, end]` that no
+//!    tag still reads;
+//! 2. changelog files added by `[earliest, end)`;
+//! 3. manifest lists, manifests, index manifests, index files, statistics and
+//!    reassign plans of `[earliest, end)` that neither `end` nor a tag in the
+//!    range still references;
+//! 4. the snapshot files themselves, then the EARLIEST hint moves to `end`.
+//!
+//! Snapshot files go last, so an interrupted run leaves every remaining
+//! snapshot readable and a later run finishes the job.
+
+use crate::spec::{CoreOptions, Snapshot};
+use crate::table::snapshot_deletion::{read_long_lived_changelogs, DataFileKey, SnapshotDeletion};
+use crate::table::{BranchManager, Table};
+use crate::{Error, Result};
+use futures::{stream, StreamExt, TryStreamExt};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+use std::time::{SystemTime, UNIX_EPOCH};
+
+const SNAPSHOT_READ_CONCURRENCY: usize = 16;
+
+/// Retention rules for one expiration run. Java `ExpireConfig` (snapshot part).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+struct ExpireConfig {
+    retain_max: i32,
+    retain_min: i32,
+    /// Snapshots whose successor was committed at or after this time are kept.
+    older_than_millis: i64,
+    max_deletes: i32,
+}
+
+/// Expire old snapshots of a table and delete the files only they reference.
+///
+/// Defaults come from the table options `snapshot.num-retained.min`,
+/// `snapshot.num-retained.max`, `snapshot.time-retained` and
+/// `snapshot.expire.limit`; each can be overridden per run, like the arguments
+/// of Java's `expire_snapshots` procedure.
+pub struct ExpireSnapshots<'a> {
+    table: &'a Table,
+    retain_max: Option<i32>,
+    retain_min: Option<i32>,
+    older_than_millis: Option<i64>,
+    max_deletes: Option<i32>,
+    current_time_millis: Option<i64>,
+}
+
+impl<'a> ExpireSnapshots<'a> {
+    pub(crate) fn new(table: &'a Table) -> Self {
+        Self {
+            table,
+            retain_max: None,
+            retain_min: None,
+            older_than_millis: None,
+            max_deletes: None,
+            current_time_millis: None,
+        }
+    }
+
+    /// Maximum number of completed snapshots to retain.
+    pub fn with_retain_max(&mut self, retain_max: i32) -> &mut Self {
+        self.retain_max = Some(retain_max);
+        self
+    }
+
+    /// Minimum number of completed snapshots to retain.
+    pub fn with_retain_min(&mut self, retain_min: i32) -> &mut Self {
+        self.retain_min = Some(retain_min);
+        self
+    }
+
+    /// Expire snapshots older than this epoch-millisecond timestamp instead of
+    /// `snapshot.time-retained`.
+    pub fn with_older_than_millis(&mut self, older_than_millis: i64) -> &mut Self {
+        self.older_than_millis = Some(older_than_millis);
+        self
+    }
+
+    /// Maximum number of snapshots to expire in this run.
+    pub fn with_max_deletes(&mut self, max_deletes: i32) -> &mut Self {
+        self.max_deletes = Some(max_deletes);
+        self
+    }
+
+    #[cfg(test)]
+    pub(crate) fn with_current_time_millis(&mut self, current_time_millis: i64) -> &mut Self {
+        self.current_time_millis = Some(current_time_millis);
+        self
+    }
+
+    fn config(&self) -> Result<ExpireConfig> {
+        let core = CoreOptions::new(self.table.schema().options());
+        let retain_max = match self.retain_max {
+            Some(value) => positive("retain_max", value)?,
+            None => core.snapshot_num_retained_max()?,
+        };
+        let retain_min = match self.retain_min {
+            Some(value) => positive("retain_min", value)?,
+            None => core.snapshot_num_retained_min()?,
+        };
+        if retain_max < retain_min {
+            return Err(Error::DataInvalid {
+                message: format!(
+                    "retainMax ({retain_max}) must not be less than retainMin ({retain_min})."
+                ),
+                source: None,
+            });
+        }
+        let max_deletes = match self.max_deletes {
+            Some(value) => positive("max_deletes", value)?,
+            None => core.snapshot_expire_limit()?,
+        };
+        let older_than_millis = match self.older_than_millis {
+            Some(value) => value,
+            None => {
+                let now = self.current_time_millis.unwrap_or_else(current_time_millis);
+                let retained = i64::try_from(core.snapshot_time_retained_ms()?).unwrap_or(i64::MAX);
+                now.saturating_sub(retained)
+            }
+        };
+        Ok(ExpireConfig {
+            retain_max,
+            retain_min,
+            older_than_millis,
+            max_deletes,
+        })
+    }
+
+    /// Expire snapshots and return how many snapshot files were removed.
+    pub async fn execute(&self) -> Result<usize> {
+        self.table.ensure_not_branch_reference_for_write()?;
+        let config = self.config()?;
+
+        let snapshot_manager = self.table.snapshot_manager();
+        let Some(latest) = snapshot_manager.get_latest_snapshot().await? else {
+            return Ok(0);
+        };
+        let latest = latest.id();
+        let Some(earliest) = snapshot_manager.earliest_snapshot_id().await? else {
+            return Ok(0);
+        };
+
+        // The oldest snapshot `snapshot.num-retained.max` lets us keep.
+        let min = (latest - i64::from(config.retain_max) + 1).max(earliest);
+        // `snapshot.num-retained.min` protects the newest snapshots.
+        let mut max_exclusive = latest - i64::from(config.retain_min) + 1;
+        // A snapshot a consumer still reads from cannot be deleted.
+        let consumers = self.table.consumer_manager().list_all().await?;
+        if let Some(min_next) = consumers.iter().map(|(_, next)| *next).min() {
+            max_exclusive = max_exclusive.min(min_next);
+        }
+        // `snapshot.expire.limit` bounds one run.
+        max_exclusive = max_exclusive.min(earliest.saturating_add(i64::from(config.max_deletes)));
+
+        for id in min..max_exclusive {
+            // A snapshot expires only once its successor has also outlived the
+            // retention time.
+            if let Some(next) = snapshot_manager.try_get_snapshot(id + 1).await? {
+                if config.older_than_millis <= next.time_millis() as i64 {
+                    return self.expire_until(earliest, id).await;
+                }
+            }
+        }
+        self.expire_until(earliest, max_exclusive).await
+    }
+
+    async fn expire_until(&self, earliest: i64, end_exclusive: i64) -> Result<usize> {
+        let snapshot_manager = self.table.snapshot_manager();
+        if end_exclusive <= earliest {
+            // Nothing expires; record the earliest snapshot so later runs find
+            // it without listing the directory.
+            if !snapshot_manager.earliest_hint_exists().await? {
+                snapshot_manager.write_earliest_hint(earliest).await?;
+            }
+            return Ok(0);
+        }
+
+        // Futures are built eagerly: a borrowing closure inside the stream
+        // would make callers' futures lose `Send`.
+        let reads = (earliest..=end_exclusive)
+            .map(|id| snapshot_manager.try_get_snapshot(id))
+            .collect::<Vec<_>>();
+        let snapshots_including_end = stream::iter(reads)
+            .buffered(SNAPSHOT_READ_CONCURRENCY)
+            .try_collect::<Vec<_>>()
+            .await?
+            .into_iter()
+            .flatten()
+            .collect::<Vec<_>>();
+        let Some(first) = snapshots_including_end.first() else {
+            return Ok(0);
+        };
+        let begin_inclusive = first.id();
+        let snapshots_excluding_end = snapshots_including_end
+            .iter()
+            .filter(|snapshot| snapshot.id() != end_exclusive)
+            .collect::<Vec<_>>();
+
+        let tagged = self.tagged_snapshots().await?;
+        let deletion = SnapshotDeletion::new(self.table)?;
+        // Branches and long-lived changelogs share files with this history.
+        // Read everything they reference before deleting anything; if that is
+        // impossible, the run stops with nothing changed.
+        let external = self.external_owners(&deletion).await?;
+        // Their index and changelog manifests too: nothing may be deleted
+        // before every owner outside main's history has been read.
+        let external_snapshots = external.snapshots.iter().collect::<Vec<_>>();
+        let external_skipping = deletion
+            .manifest_skipping_set(&external_snapshots, true)
+            .await?;
+
+        // Data files deleted by a snapshot are unused from that snapshot on,
+        // so the range is (begin, end].
+        let data_snapshots = snapshots_including_end
+            .iter()
+            .filter(|snapshot| snapshot.id() != begin_inclusive)
+            .collect::<Vec<_>>();
+        self.clean_data_files(&deletion, &data_snapshots, &tagged, &external.data_files)
+            .await;
+
+        let mut changelog_files = Vec::new();
+        for snapshot in &snapshots_excluding_end {
+            let still_read = snapshot
+                .changelog_manifest_list()
+                .is_some_and(|list| external.changelog_lists.contains(list));
+            if !still_read {
+                changelog_files.extend(deletion.changelog_files(snapshot).await);
+            }
+        }
+        deletion.delete_quietly(changelog_files).await;
+
+        let last = snapshots_including_end
+            .last()
+            .expect("checked non-empty above");
+        if last.id() != end_exclusive {
+            // The end snapshot is gone, so nothing protects the manifests it
+            // would share; stop before deleting any of them.
+            return Ok(0);
+        }
+        let mut skipping_snapshots = find_skipping_tags(&tagged, begin_inclusive, end_exclusive);
+        skipping_snapshots.push(last);
+        match deletion
+            .manifest_skipping_set(&skipping_snapshots, false)
+            .await
+        {
+            Ok(mut skipping) => {
+                skipping.extend(external_skipping);
+                let mut manifest_files = Vec::new();
+                for snapshot in &snapshots_excluding_end {
+                    manifest_files.extend(
+                        deletion
+                            .unused_manifest_paths(snapshot, &mut skipping)
+                            .await?,
+                    );
+                }
+                deletion.delete_quietly(manifest_files).await;
+            }
+            Err(error) => log::info!(
+                "Skip cleaning manifest files because the skipping set cannot be built: {error}"
+            ),
+        }
+
+        for snapshot in &snapshots_excluding_end {
+            if let Err(error) = snapshot_manager.delete_snapshot(snapshot.id()).await {
+                log::warn!("Failed to delete snapshot {}: {error}", snapshot.id());
+            }
+        }
+        snapshot_manager.write_earliest_hint(end_exclusive).await?;
+        Ok(snapshots_excluding_end.len())
+    }
+
+    /// Delete the data files each snapshot's delta removes, keeping the ones
+    /// the closest earlier tag still reads. A file removed after a tag was taken
+    /// is live in that tag and in no later snapshot, so the closest earlier tag
+    /// is the only one that can still read it.
+    async fn clean_data_files(
+        &self,
+        deletion: &SnapshotDeletion,
+        snapshots: &[&Snapshot],
+        tagged: &[Snapshot],
+        external: &HashSet<DataFileKey>,
+    ) {
+        let plans = snapshots
+            .iter()
+            .map(|snapshot| deletion.plan_deleted_in_delta_manifest(snapshot))
+            .collect::<Vec<_>>();
+        let plans = stream::iter(plans)
+            .buffered(SNAPSHOT_READ_CONCURRENCY)
+            .collect::<Vec<_>>()
+            .await;
+
+        let mut tag_files: HashMap<i64, Option<HashSet<DataFileKey>>> = HashMap::new();
+        let mut paths = Vec::new();
+        for (snapshot, plan) in snapshots.iter().zip(&plans) {
+            let Some(tag) = previous_tag(tagged, snapshot.id()) else {
+                paths.extend(plan.paths_to_delete(&[external]));
+                continue;
+            };
+            if let Entry::Vacant(slot) = tag_files.entry(tag.id()) {
+                let files = match deletion.tag_data_files(tag).await {
+                    Ok(files) => Some(files),
+                    Err(error) => {
+                        log::info!(
+                            "Failed to read the data files of tag snapshot {}: {error}",
+                            tag.id()
+                        );
+                        None
+                    }
+                };
+                slot.insert(files);
+            }
+            match &tag_files[&tag.id()] {
+                Some(files) => paths.extend(plan.paths_to_delete(&[files, external])),
+                None => log::info!(
+                    "Skip cleaning data files of snapshot {} because tag snapshot {} cannot be read",
+                    snapshot.id(),
+                    tag.id()
+                ),
+            }
+        }
+        deletion.delete_quietly(paths).await;
+    }
+
+    /// Snapshots and tags of every branch, and long-lived changelogs of main
+    /// and every branch. Unlike main's own history, their files cannot be
+    /// reasoned about from the expired range, so everything they reference is
+    /// kept.
+    async fn external_owners(&self, deletion: &SnapshotDeletion) -> Result<ExternalOwners> {
+        let file_io = self.table.file_io();
+        let branch_manager = BranchManager::new(file_io.clone(), self.table.location().to_string());
+        let mut snapshots = read_long_lived_changelogs(file_io, self.table.location()).await?;
+        for branch in branch_manager.list_all().await? {
+            let snapshot_manager = self.table.snapshot_manager().with_branch(&branch);
+            for id in snapshot_manager.list_all_ids().await? {

Review Comment:
   Fixed in 6e3242c. During the existing preflight, `external_owners` now also adds each branch's latest snapshot from the catalog-aware `snapshot_manager.get_latest_snapshot().await?`, next to the branch's filesystem snapshots, tags and long-lived changelogs. A failed lookup returns the error before the first deletion.
   
   Two tests use a REST mock catalog and a branch `b1` that has a schema but no snapshot JSON or tags on storage:
   - `test_catalog_only_branch_snapshot_is_protected`: main is overwritten to snapshot 2 while the catalog serves snapshot 1 for `b1`. Expiring main down to one snapshot keeps snapshot 1's manifest lists, the branch still reads row 1, and the catalog receives the `$branch_b1` snapshot request.
   - `test_failed_branch_catalog_lookup_changes_nothing`: the catalog answers 500 for `b1`, and the run fails with snapshots, data files and manifests unchanged.
   
   Both fail without the fix.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "zhuxiangyi (via GitHub)" <gi...@apache.org>.
zhuxiangyi commented on code in PR #965:
URL: https://github.com/apache/paimon-rust/pull/965#discussion_r4173318672


##########
crates/paimon/src/table/expire_snapshots.rs:
##########
@@ -0,0 +1,473 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Snapshot expiration.
+//!
+//! Reference: Java `ExpireSnapshotsImpl`, configured like the Flink and Spark
+//! `expire_snapshots` procedures (`ProcedureUtils.fillInSnapshotOptions`).
+//!
+//! Expiring snapshots `[earliest, end)` removes, in order:
+//! 1. data files deleted by the delta manifests of `(earliest, end]` that no
+//!    tag still reads;
+//! 2. changelog files added by `[earliest, end)`;
+//! 3. manifest lists, manifests, index manifests, index files, statistics and
+//!    reassign plans of `[earliest, end)` that neither `end` nor a tag in the
+//!    range still references;
+//! 4. the snapshot files themselves, then the EARLIEST hint moves to `end`.
+//!
+//! Snapshot files go last, so an interrupted run leaves every remaining
+//! snapshot readable and a later run finishes the job.
+
+use crate::spec::{CoreOptions, Snapshot};
+use crate::table::snapshot_deletion::{read_long_lived_changelogs, DataFileKey, SnapshotDeletion};
+use crate::table::{BranchManager, Table};
+use crate::{Error, Result};
+use futures::{stream, StreamExt, TryStreamExt};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+use std::time::{SystemTime, UNIX_EPOCH};
+
+const SNAPSHOT_READ_CONCURRENCY: usize = 16;
+
+/// Retention rules for one expiration run. Java `ExpireConfig` (snapshot part).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+struct ExpireConfig {
+    retain_max: i32,
+    retain_min: i32,
+    /// Snapshots whose successor was committed at or after this time are kept.
+    older_than_millis: i64,
+    max_deletes: i32,
+}
+
+/// Expire old snapshots of a table and delete the files only they reference.
+///
+/// Defaults come from the table options `snapshot.num-retained.min`,
+/// `snapshot.num-retained.max`, `snapshot.time-retained` and
+/// `snapshot.expire.limit`; each can be overridden per run, like the arguments
+/// of Java's `expire_snapshots` procedure.
+pub struct ExpireSnapshots<'a> {
+    table: &'a Table,
+    retain_max: Option<i32>,
+    retain_min: Option<i32>,
+    older_than_millis: Option<i64>,
+    max_deletes: Option<i32>,
+    current_time_millis: Option<i64>,
+}
+
+impl<'a> ExpireSnapshots<'a> {
+    pub(crate) fn new(table: &'a Table) -> Self {
+        Self {
+            table,
+            retain_max: None,
+            retain_min: None,
+            older_than_millis: None,
+            max_deletes: None,
+            current_time_millis: None,
+        }
+    }
+
+    /// Maximum number of completed snapshots to retain.
+    pub fn with_retain_max(&mut self, retain_max: i32) -> &mut Self {
+        self.retain_max = Some(retain_max);
+        self
+    }
+
+    /// Minimum number of completed snapshots to retain.
+    pub fn with_retain_min(&mut self, retain_min: i32) -> &mut Self {
+        self.retain_min = Some(retain_min);
+        self
+    }
+
+    /// Expire snapshots older than this epoch-millisecond timestamp instead of
+    /// `snapshot.time-retained`.
+    pub fn with_older_than_millis(&mut self, older_than_millis: i64) -> &mut Self {
+        self.older_than_millis = Some(older_than_millis);
+        self
+    }
+
+    /// Maximum number of snapshots to expire in this run.
+    pub fn with_max_deletes(&mut self, max_deletes: i32) -> &mut Self {
+        self.max_deletes = Some(max_deletes);
+        self
+    }
+
+    #[cfg(test)]
+    pub(crate) fn with_current_time_millis(&mut self, current_time_millis: i64) -> &mut Self {
+        self.current_time_millis = Some(current_time_millis);
+        self
+    }
+
+    fn config(&self) -> Result<ExpireConfig> {
+        let core = CoreOptions::new(self.table.schema().options());
+        let retain_max = match self.retain_max {
+            Some(value) => positive("retain_max", value)?,
+            None => core.snapshot_num_retained_max()?,
+        };
+        let retain_min = match self.retain_min {
+            Some(value) => positive("retain_min", value)?,
+            None => core.snapshot_num_retained_min()?,
+        };
+        if retain_max < retain_min {
+            return Err(Error::DataInvalid {
+                message: format!(
+                    "retainMax ({retain_max}) must not be less than retainMin ({retain_min})."
+                ),
+                source: None,
+            });
+        }
+        let max_deletes = match self.max_deletes {
+            Some(value) => positive("max_deletes", value)?,
+            None => core.snapshot_expire_limit()?,
+        };
+        let older_than_millis = match self.older_than_millis {
+            Some(value) => value,
+            None => {
+                let now = self.current_time_millis.unwrap_or_else(current_time_millis);
+                let retained = i64::try_from(core.snapshot_time_retained_ms()?).unwrap_or(i64::MAX);
+                now.saturating_sub(retained)
+            }
+        };
+        Ok(ExpireConfig {
+            retain_max,
+            retain_min,
+            older_than_millis,
+            max_deletes,
+        })
+    }
+
+    /// Expire snapshots and return how many snapshot files were removed.
+    pub async fn execute(&self) -> Result<usize> {
+        self.table.ensure_not_branch_reference_for_write()?;
+        let config = self.config()?;
+
+        let snapshot_manager = self.table.snapshot_manager();
+        let Some(latest) = snapshot_manager.get_latest_snapshot().await? else {
+            return Ok(0);
+        };
+        let latest = latest.id();
+        let Some(earliest) = snapshot_manager.earliest_snapshot_id().await? else {
+            return Ok(0);
+        };
+
+        // The oldest snapshot `snapshot.num-retained.max` lets us keep.
+        let min = (latest - i64::from(config.retain_max) + 1).max(earliest);
+        // `snapshot.num-retained.min` protects the newest snapshots.
+        let mut max_exclusive = latest - i64::from(config.retain_min) + 1;
+        // A snapshot a consumer still reads from cannot be deleted.
+        let consumers = self.table.consumer_manager().list_all().await?;
+        if let Some(min_next) = consumers.iter().map(|(_, next)| *next).min() {
+            max_exclusive = max_exclusive.min(min_next);
+        }
+        // `snapshot.expire.limit` bounds one run.
+        max_exclusive = max_exclusive.min(earliest.saturating_add(i64::from(config.max_deletes)));
+
+        for id in min..max_exclusive {
+            // A snapshot expires only once its successor has also outlived the
+            // retention time.
+            if let Some(next) = snapshot_manager.try_get_snapshot(id + 1).await? {
+                if config.older_than_millis <= next.time_millis() as i64 {
+                    return self.expire_until(earliest, id).await;
+                }
+            }
+        }
+        self.expire_until(earliest, max_exclusive).await
+    }
+
+    async fn expire_until(&self, earliest: i64, end_exclusive: i64) -> Result<usize> {
+        let snapshot_manager = self.table.snapshot_manager();
+        if end_exclusive <= earliest {
+            // Nothing expires; record the earliest snapshot so later runs find
+            // it without listing the directory.
+            if !snapshot_manager.earliest_hint_exists().await? {
+                snapshot_manager.write_earliest_hint(earliest).await?;
+            }
+            return Ok(0);
+        }
+
+        // Futures are built eagerly: a borrowing closure inside the stream
+        // would make callers' futures lose `Send`.
+        let reads = (earliest..=end_exclusive)
+            .map(|id| snapshot_manager.try_get_snapshot(id))
+            .collect::<Vec<_>>();
+        let snapshots_including_end = stream::iter(reads)
+            .buffered(SNAPSHOT_READ_CONCURRENCY)
+            .try_collect::<Vec<_>>()
+            .await?
+            .into_iter()
+            .flatten()
+            .collect::<Vec<_>>();
+        let Some(first) = snapshots_including_end.first() else {
+            return Ok(0);
+        };
+        let begin_inclusive = first.id();
+        let snapshots_excluding_end = snapshots_including_end
+            .iter()
+            .filter(|snapshot| snapshot.id() != end_exclusive)
+            .collect::<Vec<_>>();
+
+        let tagged = self.tagged_snapshots().await?;
+        let deletion = SnapshotDeletion::new(self.table)?;
+        // Branches and long-lived changelogs share files with this history.
+        // Read everything they reference before deleting anything; if that is
+        // impossible, the run stops with nothing changed.
+        let external = self.external_owners(&deletion).await?;
+
+        // Data files deleted by a snapshot are unused from that snapshot on,
+        // so the range is (begin, end].
+        let data_snapshots = snapshots_including_end
+            .iter()
+            .filter(|snapshot| snapshot.id() != begin_inclusive)
+            .collect::<Vec<_>>();
+        self.clean_data_files(&deletion, &data_snapshots, &tagged, &external.data_files)
+            .await;
+
+        let mut changelog_files = Vec::new();
+        for snapshot in &snapshots_excluding_end {
+            let still_read = snapshot
+                .changelog_manifest_list()
+                .is_some_and(|list| external.changelog_lists.contains(list));
+            if !still_read {
+                changelog_files.extend(deletion.changelog_files(snapshot).await);
+            }
+        }
+        deletion.delete_quietly(changelog_files).await;
+
+        let last = snapshots_including_end
+            .last()
+            .expect("checked non-empty above");
+        if last.id() != end_exclusive {
+            // The end snapshot is gone, so nothing protects the manifests it
+            // would share; stop before deleting any of them.
+            return Ok(0);
+        }
+        let mut skipping_snapshots = find_skipping_tags(&tagged, begin_inclusive, end_exclusive);
+        skipping_snapshots.push(last);
+        let external_snapshots = external.snapshots.iter().collect::<Vec<_>>();
+        let skipping = match deletion
+            .manifest_skipping_set(&skipping_snapshots, false)
+            .await
+        {
+            Ok(mut skipping) => {
+                skipping.extend(
+                    deletion
+                        .manifest_skipping_set(&external_snapshots, true)

Review Comment:
   Fixed in f7ba2d3. The skipping set of the external owners reads their index and changelog manifests. It is now built right after the owners are read, before any data or changelog file is deleted. A run that cannot read one of them now fails with nothing changed.
   
   - `test_unreadable_branch_index_manifest_changes_nothing` follows your probe: build a global index, branch from the indexed snapshot, drop main's index, overwrite twice, then corrupt only the branch-owned index manifest. The call fails, and every snapshot and data file remains.
   - `test_unreadable_external_changelog_manifest_changes_nothing` covers an unreadable changelog manifest of a long-lived changelog.
   
   Both tests fail if the set is built after the deletions again.
   



##########
crates/paimon/src/table/snapshot_deletion.rs:
##########
@@ -0,0 +1,500 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Planning and deleting the files of expired snapshots.
+//!
+//! Reference: Java `FileDeletionBase` and `SnapshotDeletion`.
+//!
+//! Every read failure makes the plan delete less, never more: an unreadable
+//! delta manifest cancels that snapshot's data-file deletion, and a skipping
+//! set that cannot be built cancels manifest deletion. A file left behind is
+//! an orphan that orphan-file cleanup can remove later; a referenced file
+//! deleted by mistake is data loss.
+
+use crate::io::FileIO;
+use crate::spec::{
+    bucket_path, BinaryRow, CoreOptions, FileKind, IndexManifest, Manifest, ManifestEntry,
+    ManifestFileMeta, ManifestList, PartitionComputer, Snapshot,
+};
+use crate::table::index_file_path::{
+    committed_index_file_path, resolve_legacy_deletion_vector_entries,
+};
+use crate::table::{SnapshotManager, Table};
+use crate::Result;
+use futures::{stream, StreamExt, TryStreamExt};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+
+const FILE_OPERATION_CONCURRENCY: usize = 16;
+const STATISTICS_DIR: &str = "statistics";
+/// Java `SerializationAssignment.PLAN_FILE_PROPERTY`.
+const REASSIGN_PLAN_FILE_PROPERTY: &str = "row-id-reassign.plan";
+/// Java `SerializationAssignment.REASSIGN_SNAPSHOT_ID`.
+const REASSIGN_SNAPSHOT_ID_PROPERTY: &str = "reassign-snapshot-id";
+
+/// A data file within a bucket: `(partition, bucket, file name)`. File names
+/// are unique within a bucket, so this is how Java's `containsDataFile`
+/// matches entries across snapshots.
+pub(crate) type DataFileKey = (Vec<u8>, i32, String);
+
+/// Data files that a snapshot's delta manifests delete, with the physical
+/// paths (data file plus extra files) to remove for each.
+#[derive(Debug, Default)]
+pub(crate) struct DataFileDeletionPlan {
+    candidates: Vec<(DataFileKey, Vec<String>)>,
+}
+
+impl DataFileDeletionPlan {
+    /// Paths of candidates that none of `retained` protects.
+    pub(crate) fn paths_to_delete(&self, retained: &[&HashSet<DataFileKey>]) -> Vec<String> {
+        self.candidates
+            .iter()
+            .filter(|(key, _)| !retained.iter().any(|retained| retained.contains(key)))
+            .flat_map(|(_, paths)| paths.iter().cloned())
+            .collect()
+    }
+}
+
+pub(crate) struct SnapshotDeletion {
+    table: Table,
+    file_io: FileIO,
+    table_location: String,
+    snapshot_manager: SnapshotManager,
+    partition_computer: Option<PartitionComputer>,
+    index_file_in_data_file_dir: bool,
+}
+
+impl SnapshotDeletion {
+    pub(crate) fn new(table: &Table) -> Result<Self> {
+        let schema = table.schema();
+        let core_options = CoreOptions::new(schema.options());
+        let partition_computer = if schema.partition_keys().is_empty() {
+            None
+        } else {
+            Some(PartitionComputer::new(
+                schema.partition_keys(),
+                schema.fields(),
+                core_options.partition_default_name(),
+                core_options.legacy_partition_name(),
+            )?)
+        };
+        Ok(Self {
+            table: table.clone(),
+            file_io: table.file_io().clone(),
+            table_location: table.location().trim_end_matches('/').to_string(),
+            snapshot_manager: table.snapshot_manager(),
+            partition_computer,
+            index_file_in_data_file_dir: core_options.index_file_in_data_file_dir(),
+        })
+    }
+
+    fn bucket_path(&self, partition: &[u8], bucket: i32) -> Result<String> {
+        let partition = match self.partition_computer {
+            // An unpartitioned table never decodes its (empty) partition blob.
+            None => BinaryRow::new(0),
+            Some(_) => BinaryRow::from_serialized_bytes(partition)?,
+        };
+        bucket_path(
+            &self.table_location,

Review Comment:
   Fixed in f7ba2d3. Bucket directories now come from `Table::data_file_location()`, as for writers, readers and commit cleanup. That covers data files, their sidecars, and bucket-local index files. Manifests, statistics and global index files stay under the table location.
   
   Covered in the core tests for a relative (`data`) and an absolute `data-file.path-directory`, and end to end through `CALL sys.expire_snapshots` in `test_expire_snapshots_procedure_with_data_directory`. Each test fails if the table location is used again.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "zhuxiangyi (via GitHub)" <gi...@apache.org>.
zhuxiangyi commented on PR #965:
URL: https://github.com/apache/paimon-rust/pull/965#issuecomment-5969530987

   Round 2 is addressed in f7ba2d3; details are in the inline replies. The branch is rebased onto main c41ce20.
   
   The commit ids in my previous reply predate two rebases. e1d75d2 is now 74741ed, and the new base is c41ce20 rather than 2d0dda1.
   
   `cargo test -p paimon --all-targets --features fulltext,vortex`, the DataFusion procedure tests and clippy with `-D warnings` pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "zhuxiangyi (via GitHub)" <gi...@apache.org>.
zhuxiangyi commented on code in PR #965:
URL: https://github.com/apache/paimon-rust/pull/965#discussion_r4178227748


##########
crates/paimon/src/table/snapshot_deletion.rs:
##########
@@ -0,0 +1,504 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Planning and deleting the files of expired snapshots.
+//!
+//! Reference: Java `FileDeletionBase` and `SnapshotDeletion`.
+//!
+//! Every read failure makes the plan delete less, never more: an unreadable
+//! delta manifest cancels that snapshot's data-file deletion, and a skipping
+//! set that cannot be built cancels manifest deletion. A file left behind is
+//! an orphan that orphan-file cleanup can remove later; a referenced file
+//! deleted by mistake is data loss.
+
+use crate::io::FileIO;
+use crate::spec::{
+    bucket_path, BinaryRow, CoreOptions, FileKind, IndexManifest, Manifest, ManifestEntry,
+    ManifestFileMeta, ManifestList, PartitionComputer, Snapshot,
+};
+use crate::table::index_file_path::{
+    committed_index_file_path, resolve_legacy_deletion_vector_entries,

Review Comment:
   Fixed in 5e9cbad. I rebased onto main 8c3527b and reproduced the E0432 first. #1016 removed the legacy deletion-vector location fallback from reads altogether, not only that helper, so expiration now resolves index files only through `committed_index_file_path`, the layout reads use today. The round-1 legacy-location test is dropped, because that layout is no longer readable on main.
   
   The merge with current main then compiles. The full core suite, the DataFusion procedure tests and clippy with `-D warnings` pass on top of 8c3527b.
   



##########
crates/integrations/datafusion/src/procedures.rs:
##########
@@ -1214,6 +1224,82 @@ async fn proc_list_policies(
     )
 }
 
+/// `CALL sys.expire_snapshots`, with the arguments of Java's Flink and Spark
+/// `ExpireSnapshotsProcedure`. `options` are dynamic table options applied for
+/// this call only (for example `snapshot.time-retained`).
+async fn proc_expire_snapshots(
+    ctx: &SessionContext,
+    catalog: &Arc<dyn Catalog>,
+    catalog_name: &str,
+    args: &HashMap<String, String>,
+) -> DFResult<DataFrame> {
+    let mut table = get_table(catalog, catalog_name, args).await?;
+    if let Some(options) = args.get("options") {
+        table = table.copy_with_options(parse_key_value_options(options)?);
+    }
+    let mut expire = table.new_expire_snapshots();
+    if let Some(value) = optional_i32_arg(args, "retain_max")? {
+        expire.with_retain_max(value);
+    }
+    if let Some(value) = optional_i32_arg(args, "retain_min")? {
+        expire.with_retain_min(value);
+    }
+    if let Some(value) = optional_i32_arg(args, "max_deletes")? {
+        expire.with_max_deletes(value);
+    }
+    if let Some(older_than) = args.get("older_than").filter(|v| !v.trim().is_empty()) {
+        expire.with_older_than_millis(parse_older_than(older_than)?);
+    }
+    let deleted = expire.execute().await.map_err(to_datafusion_error)?;
+    let deleted = i32::try_from(deleted).unwrap_or(i32::MAX);
+
+    let schema = Arc::new(Schema::new(vec![Field::new(
+        "deleted_snapshots_count",
+        ArrowDataType::Int32,
+        false,
+    )]));
+    let batch = RecordBatch::try_new(schema, vec![Arc::new(Int32Array::from(vec![deleted]))])?;
+    ctx.read_batch(batch)
+}
+
+fn optional_i32_arg(args: &HashMap<String, String>, name: &str) -> DFResult<Option<i32>> {
+    args.get(name)
+        .map(|value| {
+            value.trim().parse::<i32>().map_err(|_| {
+                DataFusionError::Plan(format!("Invalid integer for '{name}': '{value}'"))
+            })
+        })
+        .transpose()
+}
+
+/// `older_than` as epoch milliseconds, or as a timestamp such as
+/// `2024-01-01 12:00:00` in the session's local time zone, which is how Java
+/// (`DateTimeUtils.parseTimestampData` with the default time zone) reads it.
+fn parse_older_than(value: &str) -> DFResult<i64> {
+    let value = value.trim();
+    if let Ok(millis) = value.parse::<i64>() {
+        return Ok(millis);
+    }
+    let naive = ["%Y-%m-%d %H:%M:%S%.f", "%Y-%m-%dT%H:%M:%S%.f"]
+        .iter()
+        .find_map(|format| NaiveDateTime::parse_from_str(value, format).ok())
+        .or_else(|| {
+            NaiveDate::parse_from_str(value, "%Y-%m-%d")
+                .ok()
+                .and_then(|date| date.and_hms_opt(0, 0, 0))
+        })
+        .ok_or_else(|| DataFusionError::Plan(format!("Invalid older_than timestamp: '{value}'")))?;
+    Local

Review Comment:
   Fixed in 22d9e7e. `older_than` now goes through `spec::parse_local_timestamp_millis`. That is main's Java-compatible `scan.timestamp` parser (jiff's compatible disambiguation, as in `LocalDateTime.atZone`), generalized to take an option name. A time in a gap moves forward, and one in a fold takes the earlier instant. Epoch milliseconds are still accepted.
   
   `test_parse_older_than_follows_java_across_dst` runs the parser in a child process pinned to `TZ=America/New_York`, like the existing `scan.timestamp` test. `2024-03-10 02:30:00` gives 1710055800000 (03:30 EDT), and `2024-11-03 01:30:00` gives the earlier 1730611800000. On the old code it fails with your error message.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #965:
URL: https://github.com/apache/paimon-rust/pull/965#discussion_r4177291194


##########
crates/paimon/src/table/snapshot_deletion.rs:
##########
@@ -0,0 +1,504 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Planning and deleting the files of expired snapshots.
+//!
+//! Reference: Java `FileDeletionBase` and `SnapshotDeletion`.
+//!
+//! Every read failure makes the plan delete less, never more: an unreadable
+//! delta manifest cancels that snapshot's data-file deletion, and a skipping
+//! set that cannot be built cancels manifest deletion. A file left behind is
+//! an orphan that orphan-file cleanup can remove later; a referenced file
+//! deleted by mistake is data loss.
+
+use crate::io::FileIO;
+use crate::spec::{
+    bucket_path, BinaryRow, CoreOptions, FileKind, IndexManifest, Manifest, ManifestEntry,
+    ManifestFileMeta, ManifestList, PartitionComputer, Snapshot,
+};
+use crate::table::index_file_path::{
+    committed_index_file_path, resolve_legacy_deletion_vector_entries,

Review Comment:
   [P1] Update snapshot deletion to the current main index-path API
   
   The merge is textually clean, but this newly added module imports resolve_legacy_deletion_vector_entries, which current main no longer provides in table::index_file_path after #1016. On the automatic merge with bede6650, the core test build fails with E0432: unresolved import crate::table::index_file_path::resolve_legacy_deletion_vector_entries. The passing head tests use the older foundation and do not establish that this PR can be merged now. Rebase/adapt the shared snapshot deletion code to current index-file resolution and rerun the merged core/SQL tests before production use.



##########
crates/integrations/datafusion/src/procedures.rs:
##########
@@ -1214,6 +1224,82 @@ async fn proc_list_policies(
     )
 }
 
+/// `CALL sys.expire_snapshots`, with the arguments of Java's Flink and Spark
+/// `ExpireSnapshotsProcedure`. `options` are dynamic table options applied for
+/// this call only (for example `snapshot.time-retained`).
+async fn proc_expire_snapshots(
+    ctx: &SessionContext,
+    catalog: &Arc<dyn Catalog>,
+    catalog_name: &str,
+    args: &HashMap<String, String>,
+) -> DFResult<DataFrame> {
+    let mut table = get_table(catalog, catalog_name, args).await?;
+    if let Some(options) = args.get("options") {
+        table = table.copy_with_options(parse_key_value_options(options)?);
+    }
+    let mut expire = table.new_expire_snapshots();
+    if let Some(value) = optional_i32_arg(args, "retain_max")? {
+        expire.with_retain_max(value);
+    }
+    if let Some(value) = optional_i32_arg(args, "retain_min")? {
+        expire.with_retain_min(value);
+    }
+    if let Some(value) = optional_i32_arg(args, "max_deletes")? {
+        expire.with_max_deletes(value);
+    }
+    if let Some(older_than) = args.get("older_than").filter(|v| !v.trim().is_empty()) {
+        expire.with_older_than_millis(parse_older_than(older_than)?);
+    }
+    let deleted = expire.execute().await.map_err(to_datafusion_error)?;
+    let deleted = i32::try_from(deleted).unwrap_or(i32::MAX);
+
+    let schema = Arc::new(Schema::new(vec![Field::new(
+        "deleted_snapshots_count",
+        ArrowDataType::Int32,
+        false,
+    )]));
+    let batch = RecordBatch::try_new(schema, vec![Arc::new(Int32Array::from(vec![deleted]))])?;
+    ctx.read_batch(batch)
+}
+
+fn optional_i32_arg(args: &HashMap<String, String>, name: &str) -> DFResult<Option<i32>> {
+    args.get(name)
+        .map(|value| {
+            value.trim().parse::<i32>().map_err(|_| {
+                DataFusionError::Plan(format!("Invalid integer for '{name}': '{value}'"))
+            })
+        })
+        .transpose()
+}
+
+/// `older_than` as epoch milliseconds, or as a timestamp such as
+/// `2024-01-01 12:00:00` in the session's local time zone, which is how Java
+/// (`DateTimeUtils.parseTimestampData` with the default time zone) reads it.
+fn parse_older_than(value: &str) -> DFResult<i64> {
+    let value = value.trim();
+    if let Ok(millis) = value.parse::<i64>() {
+        return Ok(millis);
+    }
+    let naive = ["%Y-%m-%d %H:%M:%S%.f", "%Y-%m-%dT%H:%M:%S%.f"]
+        .iter()
+        .find_map(|format| NaiveDateTime::parse_from_str(value, format).ok())
+        .or_else(|| {
+            NaiveDate::parse_from_str(value, "%Y-%m-%d")
+                .ok()
+                .and_then(|date| date.and_hms_opt(0, 0, 0))
+        })
+        .ok_or_else(|| DataFusionError::Plan(format!("Invalid older_than timestamp: '{value}'")))?;
+    Local

Review Comment:
   [P2] Match Java's forward resolution of DST gap timestamps
   
   With TZ=America/New_York, CALL sys.expire_snapshots(table => 'test_db.t1', older_than => '2024-03-10 02:30:00', retain_min => 1) fails with 'does not exist in the local time zone'. Java ProcedureUtils uses DateTimeUtils.parseTimestampData with the default time zone; LocalDateTime.atZone resolves this gap forward to 03:30 EDT (1710055800000). This rejects a valid Java procedure input instead of expiring normally. Use the same gap-forward / fold-earlier resolution already implemented for timestamp options, and cover this input in an isolated time-zone test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #965:
URL: https://github.com/apache/paimon-rust/pull/965#discussion_r4172651769


##########
crates/paimon/src/table/expire_snapshots.rs:
##########
@@ -0,0 +1,473 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Snapshot expiration.
+//!
+//! Reference: Java `ExpireSnapshotsImpl`, configured like the Flink and Spark
+//! `expire_snapshots` procedures (`ProcedureUtils.fillInSnapshotOptions`).
+//!
+//! Expiring snapshots `[earliest, end)` removes, in order:
+//! 1. data files deleted by the delta manifests of `(earliest, end]` that no
+//!    tag still reads;
+//! 2. changelog files added by `[earliest, end)`;
+//! 3. manifest lists, manifests, index manifests, index files, statistics and
+//!    reassign plans of `[earliest, end)` that neither `end` nor a tag in the
+//!    range still references;
+//! 4. the snapshot files themselves, then the EARLIEST hint moves to `end`.
+//!
+//! Snapshot files go last, so an interrupted run leaves every remaining
+//! snapshot readable and a later run finishes the job.
+
+use crate::spec::{CoreOptions, Snapshot};
+use crate::table::snapshot_deletion::{read_long_lived_changelogs, DataFileKey, SnapshotDeletion};
+use crate::table::{BranchManager, Table};
+use crate::{Error, Result};
+use futures::{stream, StreamExt, TryStreamExt};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+use std::time::{SystemTime, UNIX_EPOCH};
+
+const SNAPSHOT_READ_CONCURRENCY: usize = 16;
+
+/// Retention rules for one expiration run. Java `ExpireConfig` (snapshot part).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+struct ExpireConfig {
+    retain_max: i32,
+    retain_min: i32,
+    /// Snapshots whose successor was committed at or after this time are kept.
+    older_than_millis: i64,
+    max_deletes: i32,
+}
+
+/// Expire old snapshots of a table and delete the files only they reference.
+///
+/// Defaults come from the table options `snapshot.num-retained.min`,
+/// `snapshot.num-retained.max`, `snapshot.time-retained` and
+/// `snapshot.expire.limit`; each can be overridden per run, like the arguments
+/// of Java's `expire_snapshots` procedure.
+pub struct ExpireSnapshots<'a> {
+    table: &'a Table,
+    retain_max: Option<i32>,
+    retain_min: Option<i32>,
+    older_than_millis: Option<i64>,
+    max_deletes: Option<i32>,
+    current_time_millis: Option<i64>,
+}
+
+impl<'a> ExpireSnapshots<'a> {
+    pub(crate) fn new(table: &'a Table) -> Self {
+        Self {
+            table,
+            retain_max: None,
+            retain_min: None,
+            older_than_millis: None,
+            max_deletes: None,
+            current_time_millis: None,
+        }
+    }
+
+    /// Maximum number of completed snapshots to retain.
+    pub fn with_retain_max(&mut self, retain_max: i32) -> &mut Self {
+        self.retain_max = Some(retain_max);
+        self
+    }
+
+    /// Minimum number of completed snapshots to retain.
+    pub fn with_retain_min(&mut self, retain_min: i32) -> &mut Self {
+        self.retain_min = Some(retain_min);
+        self
+    }
+
+    /// Expire snapshots older than this epoch-millisecond timestamp instead of
+    /// `snapshot.time-retained`.
+    pub fn with_older_than_millis(&mut self, older_than_millis: i64) -> &mut Self {
+        self.older_than_millis = Some(older_than_millis);
+        self
+    }
+
+    /// Maximum number of snapshots to expire in this run.
+    pub fn with_max_deletes(&mut self, max_deletes: i32) -> &mut Self {
+        self.max_deletes = Some(max_deletes);
+        self
+    }
+
+    #[cfg(test)]
+    pub(crate) fn with_current_time_millis(&mut self, current_time_millis: i64) -> &mut Self {
+        self.current_time_millis = Some(current_time_millis);
+        self
+    }
+
+    fn config(&self) -> Result<ExpireConfig> {
+        let core = CoreOptions::new(self.table.schema().options());
+        let retain_max = match self.retain_max {
+            Some(value) => positive("retain_max", value)?,
+            None => core.snapshot_num_retained_max()?,
+        };
+        let retain_min = match self.retain_min {
+            Some(value) => positive("retain_min", value)?,
+            None => core.snapshot_num_retained_min()?,
+        };
+        if retain_max < retain_min {
+            return Err(Error::DataInvalid {
+                message: format!(
+                    "retainMax ({retain_max}) must not be less than retainMin ({retain_min})."
+                ),
+                source: None,
+            });
+        }
+        let max_deletes = match self.max_deletes {
+            Some(value) => positive("max_deletes", value)?,
+            None => core.snapshot_expire_limit()?,
+        };
+        let older_than_millis = match self.older_than_millis {
+            Some(value) => value,
+            None => {
+                let now = self.current_time_millis.unwrap_or_else(current_time_millis);
+                let retained = i64::try_from(core.snapshot_time_retained_ms()?).unwrap_or(i64::MAX);
+                now.saturating_sub(retained)
+            }
+        };
+        Ok(ExpireConfig {
+            retain_max,
+            retain_min,
+            older_than_millis,
+            max_deletes,
+        })
+    }
+
+    /// Expire snapshots and return how many snapshot files were removed.
+    pub async fn execute(&self) -> Result<usize> {
+        self.table.ensure_not_branch_reference_for_write()?;
+        let config = self.config()?;
+
+        let snapshot_manager = self.table.snapshot_manager();
+        let Some(latest) = snapshot_manager.get_latest_snapshot().await? else {
+            return Ok(0);
+        };
+        let latest = latest.id();
+        let Some(earliest) = snapshot_manager.earliest_snapshot_id().await? else {
+            return Ok(0);
+        };
+
+        // The oldest snapshot `snapshot.num-retained.max` lets us keep.
+        let min = (latest - i64::from(config.retain_max) + 1).max(earliest);
+        // `snapshot.num-retained.min` protects the newest snapshots.
+        let mut max_exclusive = latest - i64::from(config.retain_min) + 1;
+        // A snapshot a consumer still reads from cannot be deleted.
+        let consumers = self.table.consumer_manager().list_all().await?;
+        if let Some(min_next) = consumers.iter().map(|(_, next)| *next).min() {
+            max_exclusive = max_exclusive.min(min_next);
+        }
+        // `snapshot.expire.limit` bounds one run.
+        max_exclusive = max_exclusive.min(earliest.saturating_add(i64::from(config.max_deletes)));
+
+        for id in min..max_exclusive {
+            // A snapshot expires only once its successor has also outlived the
+            // retention time.
+            if let Some(next) = snapshot_manager.try_get_snapshot(id + 1).await? {
+                if config.older_than_millis <= next.time_millis() as i64 {
+                    return self.expire_until(earliest, id).await;
+                }
+            }
+        }
+        self.expire_until(earliest, max_exclusive).await
+    }
+
+    async fn expire_until(&self, earliest: i64, end_exclusive: i64) -> Result<usize> {
+        let snapshot_manager = self.table.snapshot_manager();
+        if end_exclusive <= earliest {
+            // Nothing expires; record the earliest snapshot so later runs find
+            // it without listing the directory.
+            if !snapshot_manager.earliest_hint_exists().await? {
+                snapshot_manager.write_earliest_hint(earliest).await?;
+            }
+            return Ok(0);
+        }
+
+        // Futures are built eagerly: a borrowing closure inside the stream
+        // would make callers' futures lose `Send`.
+        let reads = (earliest..=end_exclusive)
+            .map(|id| snapshot_manager.try_get_snapshot(id))
+            .collect::<Vec<_>>();
+        let snapshots_including_end = stream::iter(reads)
+            .buffered(SNAPSHOT_READ_CONCURRENCY)
+            .try_collect::<Vec<_>>()
+            .await?
+            .into_iter()
+            .flatten()
+            .collect::<Vec<_>>();
+        let Some(first) = snapshots_including_end.first() else {
+            return Ok(0);
+        };
+        let begin_inclusive = first.id();
+        let snapshots_excluding_end = snapshots_including_end
+            .iter()
+            .filter(|snapshot| snapshot.id() != end_exclusive)
+            .collect::<Vec<_>>();
+
+        let tagged = self.tagged_snapshots().await?;
+        let deletion = SnapshotDeletion::new(self.table)?;
+        // Branches and long-lived changelogs share files with this history.
+        // Read everything they reference before deleting anything; if that is
+        // impossible, the run stops with nothing changed.
+        let external = self.external_owners(&deletion).await?;
+
+        // Data files deleted by a snapshot are unused from that snapshot on,
+        // so the range is (begin, end].
+        let data_snapshots = snapshots_including_end
+            .iter()
+            .filter(|snapshot| snapshot.id() != begin_inclusive)
+            .collect::<Vec<_>>();
+        self.clean_data_files(&deletion, &data_snapshots, &tagged, &external.data_files)
+            .await;
+
+        let mut changelog_files = Vec::new();
+        for snapshot in &snapshots_excluding_end {
+            let still_read = snapshot
+                .changelog_manifest_list()
+                .is_some_and(|list| external.changelog_lists.contains(list));
+            if !still_read {
+                changelog_files.extend(deletion.changelog_files(snapshot).await);
+            }
+        }
+        deletion.delete_quietly(changelog_files).await;
+
+        let last = snapshots_including_end
+            .last()
+            .expect("checked non-empty above");
+        if last.id() != end_exclusive {
+            // The end snapshot is gone, so nothing protects the manifests it
+            // would share; stop before deleting any of them.
+            return Ok(0);
+        }
+        let mut skipping_snapshots = find_skipping_tags(&tagged, begin_inclusive, end_exclusive);
+        skipping_snapshots.push(last);
+        let external_snapshots = external.snapshots.iter().collect::<Vec<_>>();
+        let skipping = match deletion
+            .manifest_skipping_set(&skipping_snapshots, false)
+            .await
+        {
+            Ok(mut skipping) => {
+                skipping.extend(
+                    deletion
+                        .manifest_skipping_set(&external_snapshots, true)

Review Comment:
   [P2] Preflight all external owner metadata before deleting files
   
   external_owners only validates base/delta data manifests up front. External index/changelog metadata are first checked here, after clean_data_files and changelog deletion. I built an actual global index, branched from its indexed snapshot, dropped the main index, overwrote main twice, then corrupted only the branch-owned index manifest. Expiration returns the Avro read error, leaves every snapshot JSON (including snapshot4), but has already deleted snapshot4's Parquet. This violates the new guarantee that an unreadable external owner aborts with nothing changed and leaves retained residual history unreadable after a failed call. A control that preflights manifest_skipping_set(external_snapshots,true) immediately after external_owners returns the same error with all physical data preserved and the probe passes. Build/reuse that skipping set before any data/changelog mutation, and cover optional index/changelog metadata failures.



##########
crates/paimon/src/table/snapshot_deletion.rs:
##########
@@ -0,0 +1,500 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Planning and deleting the files of expired snapshots.
+//!
+//! Reference: Java `FileDeletionBase` and `SnapshotDeletion`.
+//!
+//! Every read failure makes the plan delete less, never more: an unreadable
+//! delta manifest cancels that snapshot's data-file deletion, and a skipping
+//! set that cannot be built cancels manifest deletion. A file left behind is
+//! an orphan that orphan-file cleanup can remove later; a referenced file
+//! deleted by mistake is data loss.
+
+use crate::io::FileIO;
+use crate::spec::{
+    bucket_path, BinaryRow, CoreOptions, FileKind, IndexManifest, Manifest, ManifestEntry,
+    ManifestFileMeta, ManifestList, PartitionComputer, Snapshot,
+};
+use crate::table::index_file_path::{
+    committed_index_file_path, resolve_legacy_deletion_vector_entries,
+};
+use crate::table::{SnapshotManager, Table};
+use crate::Result;
+use futures::{stream, StreamExt, TryStreamExt};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+
+const FILE_OPERATION_CONCURRENCY: usize = 16;
+const STATISTICS_DIR: &str = "statistics";
+/// Java `SerializationAssignment.PLAN_FILE_PROPERTY`.
+const REASSIGN_PLAN_FILE_PROPERTY: &str = "row-id-reassign.plan";
+/// Java `SerializationAssignment.REASSIGN_SNAPSHOT_ID`.
+const REASSIGN_SNAPSHOT_ID_PROPERTY: &str = "reassign-snapshot-id";
+
+/// A data file within a bucket: `(partition, bucket, file name)`. File names
+/// are unique within a bucket, so this is how Java's `containsDataFile`
+/// matches entries across snapshots.
+pub(crate) type DataFileKey = (Vec<u8>, i32, String);
+
+/// Data files that a snapshot's delta manifests delete, with the physical
+/// paths (data file plus extra files) to remove for each.
+#[derive(Debug, Default)]
+pub(crate) struct DataFileDeletionPlan {
+    candidates: Vec<(DataFileKey, Vec<String>)>,
+}
+
+impl DataFileDeletionPlan {
+    /// Paths of candidates that none of `retained` protects.
+    pub(crate) fn paths_to_delete(&self, retained: &[&HashSet<DataFileKey>]) -> Vec<String> {
+        self.candidates
+            .iter()
+            .filter(|(key, _)| !retained.iter().any(|retained| retained.contains(key)))
+            .flat_map(|(_, paths)| paths.iter().cloned())
+            .collect()
+    }
+}
+
+pub(crate) struct SnapshotDeletion {
+    table: Table,
+    file_io: FileIO,
+    table_location: String,
+    snapshot_manager: SnapshotManager,
+    partition_computer: Option<PartitionComputer>,
+    index_file_in_data_file_dir: bool,
+}
+
+impl SnapshotDeletion {
+    pub(crate) fn new(table: &Table) -> Result<Self> {
+        let schema = table.schema();
+        let core_options = CoreOptions::new(schema.options());
+        let partition_computer = if schema.partition_keys().is_empty() {
+            None
+        } else {
+            Some(PartitionComputer::new(
+                schema.partition_keys(),
+                schema.fields(),
+                core_options.partition_default_name(),
+                core_options.legacy_partition_name(),
+            )?)
+        };
+        Ok(Self {
+            table: table.clone(),
+            file_io: table.file_io().clone(),
+            table_location: table.location().trim_end_matches('/').to_string(),
+            snapshot_manager: table.snapshot_manager(),
+            partition_computer,
+            index_file_in_data_file_dir: core_options.index_file_in_data_file_dir(),
+        })
+    }
+
+    fn bucket_path(&self, partition: &[u8], bucket: i32) -> Result<String> {
+        let partition = match self.partition_computer {
+            // An unpartitioned table never decodes its (empty) partition blob.
+            None => BinaryRow::new(0),
+            Some(_) => BinaryRow::from_serialized_bytes(partition)?,
+        };
+        bucket_path(
+            &self.table_location,

Review Comment:
   [P2] Resolve expired buckets from the configured data directory
   
   The supported data-file.path-directory=data writes/reads `<table>/data/bucket-0`, but this helper targets `<table>/bucket-0`. A real CREATE/INSERT/OVERWRITE followed by CALL sys.expire_snapshots(retain_min=>1,retain_max=>1) removes snapshot1 metadata yet leaves its obsolete Parquet. Latest rows remain correct; physical reclamation silently fails. Use Table::data_file_location() for data and bucket-local sidecars/indexes, retaining table_location for metadata/global indexes, and cover configured-directory deletion through the public API. This shared defect is also reproduced in #966 automatic commits, #967 Python and #968 cleanup.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "zhuxiangyi (via GitHub)" <gi...@apache.org>.
zhuxiangyi commented on PR #965:
URL: https://github.com/apache/paimon-rust/pull/965#issuecomment-6049937538

   Rebased onto main 86f71e5 and addressed the review in 6e3242c. Each branch's latest snapshot, as the catalog resolves it, is now a protected owner, and a failed lookup aborts before anything is deleted. I also added a full-text index test: a dropped index's files are removed, while a rebuilt index stays searchable.
   
   `cargo test -p paimon --all-targets --features fulltext,vortex` (4373 tests), the 38 SQL procedure tests, and clippy with `-D warnings` pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(table): expire snapshots and add sys.expire_snapshots [paimon-rust]

Posted by "zhuxiangyi (via GitHub)" <gi...@apache.org>.
zhuxiangyi commented on PR #965:
URL: https://github.com/apache/paimon-rust/pull/965#issuecomment-6057363369

   Closing together with #967 and #968, following the review there: snapshot expiration and orphan cleanup delete files, and that risk isn't justified on the Rust side while Java, Flink and Spark already provide these procedures. Thanks for the detailed reviews.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org