You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "cbb330 (via GitHub)" <gi...@apache.org> on 2026/09/24 04:46:24 UTC
[I] Add creation timestamp to references [iceberg]
cbb330 opened a new issue, #18237:
URL: https://github.com/apache/iceberg/issues/18237
### Proposed Change
Add an optional, immutable `created-at-ms` field to references (branches and tags) in the table and REST specs. It records when a reference was created and is preserved as the reference moves to new snapshots.
**Motivation.** Reference expiration via `max-ref-age-ms` measures the age of the *referenced snapshot*. That intentionally prunes unused branches and tags, but it cannot express a maximum lifetime measured from reference creation:
- Each write to a branch resets its age.
- A new branch created from an older snapshot is considered old immediately.
Many branch workflows need a fixed lifetime: write-audit-publish and audit branches, automated experiment and test branches, and retention or compliance cases where a long-lived branch keeps data reachable after `main` has expired it.
Iceberg metadata currently has no reliable source for creation time. Branch creation is usually ref-only, so no snapshot records it. Branch heads move, and older metadata files are not reliably retained.
**Proposal.**
- Add optional `created-at-ms` to the reference object in table metadata and to the REST `SnapshotReference` schema.
- Set it when a reference is created, including implicit creation by a write to a missing branch.
- Preserve it through writes, target replacement, retention changes, and rename.
- For `set-snapshot-ref`, keep an existing value when an update omits the field, so clients unaware of it cannot clear it.
- Treat absence as unknown; never infer it for existing references.
- Expose it through the `refs` metadata table.
This does **not** change `max-ref-age-ms`, snapshot retention behavior, or any defaults.
**Compatibility.** Readers are unaffected. Writers that do not understand the field drop it when rewriting metadata. To keep that from erasing it, the value can't change once set: a REST update that omits it keeps the existing value, and an update that tries to change it is rejected. For catalogs where clients write metadata files directly, the proposal would require format v4 writers to preserve it.
**Context.** At LinkedIn, we run [OpenHouse](https://github.com/linkedin/openhouse), an open source control plane for lakehouse tables. Our compliance workflows—member data purges and retention cleanup—run on the `main` branch. Write-audit-publish and experiment branches are created from `main`, but those workflows and other maintenance don't run on them, and we don't want to run that maintenance on short-lived branches. A branch that outlives its intended window therefore keeps data reachable after `main` has purged it. Because we can't tell when a branch was created, we can't bound its lifetime; we have to infer the creation time from snapshot lineage, and once snapshot expiration removes that lineage, the creation time becomes unknown.
I plan to contribute the implementation.
### Proposal document
_No response_
### Specifications
- [X] Table
- [ ] View
- [X] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add creation timestamp to references [iceberg]
Posted by "cbb330 (via GitHub)" <gi...@apache.org>.
cbb330 commented on issue #18237:
URL: https://github.com/apache/iceberg/issues/18237#issuecomment-5899146646
Closing this in favor of a different approach.
While working through the design, I found that a reference's creation time doesn't bound what we actually care about. Our problem is compliant data. E.g. when a branch that split from `main` keeps rows that `main` later deletes or edits (for example, member data purges). In this case, what matters is how long a branch has been missing `main`'s changes, and that isn't necessarily how long the reference has existed, like for example:
- A branch created today from an older snapshot holds that snapshot's data.
- A branch created from another branch holds that branch's data, however recently it was created.
With that framing in mind, v3 row lineage already provides what's needed at the data level. `_row_id` and `_last_updated_sequence_number` let a maintenance job compare a branch's rows with `main`'s and find rows the branch still holds that `main` has since deleted or edited. The sequence numbers of `main`'s delete files and rewritten files, mapped to snapshot times, show how old those changes are. If that snapshot has already expired, the change is older than the retention window, which is enough for a compliance decision.
None of this needs a spec change, so we plan to handle it in OpenHouse's table maintenance.
If others need creation timestamps for lifecycle policies, independent of data compliance, I'm happy to revisit. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add creation timestamp to references [iceberg]
Posted by "cbb330 (via GitHub)" <gi...@apache.org>.
cbb330 closed issue #18237: Add creation timestamp to references
URL: https://github.com/apache/iceberg/issues/18237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Add creation timestamp to references [iceberg]
Posted by "cbb330 (via GitHub)" <gi...@apache.org>.
cbb330 closed issue #18237: Add creation timestamp to references
URL: https://github.com/apache/iceberg/issues/18237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org