You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Stephen0421 (via GitHub)" <gi...@apache.org> on 2026/04/14 07:00:48 UTC

[PR] [core] chain table support special partition expire. [paimon]

Stephen0421 opened a new pull request, #7643:
URL: https://github.com/apache/paimon/pull/7643

   ### Purpose
   This PR implements partition expiration for chain tables. Chain tables store data across snapshot
   and delta branches, where delta partitions depend on their nearest earlier snapshot partition as an
   anchor for merge-on-read. Standard partition expiration cannot be applied directly because dropping
   a snapshot partition without considering its dependent deltas would break the chain integrity.
   ### Changes
   #### New: Segment-based partition expiration (`ChainTablePartitionExpire`)
   Introduces a segment-based expiration algorithm that preserves chain integrity. A segment consists
   of one snapshot partition and all delta partitions whose time falls between that snapshot and the
   next snapshot. The segment is the atomic unit of expiration.
   Algorithm per group:
   1. Sort snapshot partitions by chain partition time.
   2. Filter to those before the cutoff (`now - partition.expiration-time`).
   3. If fewer than 2 snapshots fall before the cutoff, nothing is expired (the only one must be
      kept as anchor for its dependent deltas).
   4. The most recent snapshot before the cutoff is the anchor (kept). All earlier snapshots form
      expirable segments together with their associated delta partitions.
   5. Orphan deltas before the earliest expired snapshot are also expired.
   6. Delta partitions are dropped before snapshot partitions so that the commit pre-check passes.
   #### Refactored: `PartitionExpire` interface extraction
   Extracted `PartitionExpire` from a concrete class into an interface with three methods:
   `expire(long)`, `isValueExpiration()`, and `isValueAllExpired(Collection<BinaryRow>)`.
   The original implementation is preserved as `NormalPartitionExpire`. All existing consumers
   (Flink/Spark procedures, actions, `TableCommitImpl`, `ConflictDetection`) use only the interface
   methods — no compatibility impact.
   #### Fixed: `ChainTableCommitPreCallback` group partition awareness
   The commit pre-callback that validates snapshot partition drops was not group-partition aware.
   It used full partition comparators and triangular predicates, which could match partitions across
   different groups and produce incorrect pre/next snapshot lookups. Refactored to:
   - Filter snapshot partitions to the same group before finding pre/next.
   - Use `ChainPartitionProjector` to extract group and chain dimensions.
   - Apply group-scoped predicates for delta partition filtering.
   
   ### Tests
   - [x] Single partition key: expire segments with correct anchor retention
   - [x] No expiration when < 2 snapshots before cutoff
   - [x] Multiple segment expiration
   - [x] No expiration when no snapshots before cutoff
   - [x] Check interval prevents premature expiration
   - [x] `maxExpireNum` limits number of expired segments
   - [x] Group partition: independent expiration per group
   - [x] `isValueAllExpired`: anchor partitions not reported as expired
   - [x] `isValueAllExpired`: groups with < 2 snapshots retain all
   - [x] `isValueAllExpired`: cross-group mixed scenarios
   - [x] `isValueAllExpired`: partitions after cutoff not expired


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] chain table support special partition expire. [paimon]

Posted by "Stephen0421 (via GitHub)" <gi...@apache.org>.
Stephen0421 commented on PR #7643:
URL: https://github.com/apache/paimon/pull/7643#issuecomment-4533771017

   > Thanks for the detailed implementation and tests. I found one correctness issue in the explicit Flink action path:
   > 
   > `ExpirePartitionsAction` still builds a `PartitionExpireStrategy` from the user-provided `expireStrategy` argument and calls `FileStore.newPartitionExpire(..., expireStrategy)`. However, the chain-table branch in `AbstractFileStore.newPartitionExpire(String, FileStoreTable, Duration, Duration, PartitionExpireStrategy)` ignores that argument and always returns `newChainTablePartitionExpire(...)`.
   > 
   > This means an explicit action call with `expireStrategy = update-time` (or a custom strategy) against a chain table will silently run the chain table's values-time expiration instead of rejecting the unsupported strategy. The schema validation added in this PR catches persisted/dynamic table options, but it does not cover this action path because the action passes the strategy only as an argument to the overload.
   > 
   > Could we add the same validation in this overload before creating `ChainTablePartitionExpire` (for example, require the provided strategy to be `PartitionValuesTimeExpireStrategy`), or route the action through dynamic options so `SchemaValidation` rejects non-`values-time` consistently?
   
   Already add validation in AbstractFileStore.newPartitionExpire.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] chain table support special partition expire. [paimon]

Posted by "Stephen0421 (via GitHub)" <gi...@apache.org>.
Stephen0421 commented on PR #7643:
URL: https://github.com/apache/paimon/pull/7643#issuecomment-4533775056

   > Please rebase master.
   
   OK


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] chain table support special partition expire. [paimon]

Posted by "Stephen0421 (via GitHub)" <gi...@apache.org>.
Stephen0421 commented on PR #7643:
URL: https://github.com/apache/paimon/pull/7643#issuecomment-4326025032

   Hello @JingsongLi  , could you PTAL at this PR when you have a moment? I'd really appreciate your feedback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] chain table support special partition expire. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #7643:
URL: https://github.com/apache/paimon/pull/7643#issuecomment-4527080957

   Please rebase master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] chain table support special partition expire. [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #7643:
URL: https://github.com/apache/paimon/pull/7643


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org