You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dttung2905 (via GitHub)" <gi...@apache.org> on 2025/08/24 21:40:31 UTC
[PR] feate(table): Add time travel functionality [iceberg-go]
dttung2905 opened a new pull request, #548:
URL: https://github.com/apache/iceberg-go/pull/548
Hi team,
This PR aim to create a time travel functionality for go sdk. I took reference from iceberg-go [implementation](https://github.com/apache/iceberg-python/blob/a33cd3835136db5939e8eda9ee14a17d51ab141d/pyiceberg/table/__init__.py#L1230-L1240)
## Example for `TimeTravelScan()`
```go
// Scan data as it existed 1 day ago
oneDayAgo := time.Now().Add(-24 * time.Hour).UnixMilli()
// Basic time travel scan
scan, err := table.TimeTravelScan(oneDayAgo)
if err != nil {
log.Fatal("Time travel scan failed:", err)
}
// Time travel scan with filters and options
scanWithOptions, err := table.TimeTravelScan(oneDayAgo,
table.WithSelectedFields("id", "name", "amount"), // Select specific fields
table.WithRowFilter("amount > 10"),
table.WithLimit(10),
)
// Execute the scan
batches, err := scan.ToArrowBatches(context.Background())
if err != nil {
log.Fatal("Failed to execute scan:", err)
}
```
## Example for `TimeTravel()`
```go
twoHoursAgo := time.Now().Add(10 * time.Hour).UnixMilli()
snapshot := table.TimeTravel(twoHoursAgo, true)
fmt.Println(snapshot.SnapshotID)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feate(table): Add time travel functionality [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #548:
URL: https://github.com/apache/iceberg-go/pull/548#discussion_r2307609862
##########
table/table.go:
##########
@@ -227,6 +228,45 @@ func (t Table) doCommit(ctx context.Context, updates []Update, reqs []Requiremen
return New(t.identifier, newMeta, newLoc, t.fsF, t.cat), nil
}
+// SnapshotAsOf finds the snapshot that was current as of or right before the given timestamp.
+func (t Table) SnapshotAsOf(timestampMs int64, inclusive bool) *Snapshot {
+ entries := slices.Collect(t.metadata.SnapshotLogs())
+ for i := len(entries) - 1; i >= 0; i-- {
+ entry := entries[i]
+ if (inclusive && entry.TimestampMs <= timestampMs) || (!inclusive && entry.TimestampMs < timestampMs) {
+ return t.metadata.SnapshotByID(entry.SnapshotID)
+ }
+ }
+
+ return nil
+}
+
+// WithSnapshotAsOf creates a scan of the table as it existed at the given timestamp.
+// Note: Cannot be combined with WithSnapshotID option - time travel determines the snapshot.
+func (t Table) WithSnapshotAsOf(timestampMs int64, opts ...ScanOption) (*Scan, error) {
Review Comment:
Would this make more sense as a `ScanOption` instead of a standalone function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feate(table): Add time travel functionality [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #548:
URL: https://github.com/apache/iceberg-go/pull/548#discussion_r2298563171
##########
table/table.go:
##########
@@ -227,6 +228,45 @@ func (t Table) doCommit(ctx context.Context, updates []Update, reqs []Requiremen
return New(t.identifier, newMeta, newLoc, t.fsF, t.cat), nil
}
+// TimeTravel finds the snapshot that was current as of or right before the given timestamp.
+func (t Table) TimeTravel(timestampMs int64, inclusive bool) *Snapshot {
Review Comment:
Personally not a fan of the name here, could we do something like `SnapshotAsOf`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feate(table): Add time travel functionality [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #548:
URL: https://github.com/apache/iceberg-go/pull/548#discussion_r2298565499
##########
table/table.go:
##########
@@ -227,6 +228,45 @@ func (t Table) doCommit(ctx context.Context, updates []Update, reqs []Requiremen
return New(t.identifier, newMeta, newLoc, t.fsF, t.cat), nil
}
+// TimeTravel finds the snapshot that was current as of or right before the given timestamp.
+func (t Table) TimeTravel(timestampMs int64, inclusive bool) *Snapshot {
+ entries := slices.Collect(t.metadata.SnapshotLogs())
+ for i := len(entries) - 1; i >= 0; i-- {
+ entry := entries[i]
+ if (inclusive && entry.TimestampMs <= timestampMs) || (!inclusive && entry.TimestampMs < timestampMs) {
+ return t.metadata.SnapshotByID(entry.SnapshotID)
+ }
+ }
+
+ return nil
+}
+
+// TimeTravelScan creates a scan of the table as it existed at the given timestamp.
+// Note: Cannot be combined with WithSnapshotID option - time travel determines the snapshot.
+func (t Table) TimeTravelScan(timestampMs int64, opts ...ScanOption) (*Scan, error) {
Review Comment:
Could we just add a scan option `WithSnapshotAsOf`? Or something like that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feate(table): Add time travel functionality [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #548:
URL: https://github.com/apache/iceberg-go/pull/548#discussion_r2310431755
##########
table/table.go:
##########
@@ -227,6 +228,45 @@ func (t Table) doCommit(ctx context.Context, updates []Update, reqs []Requiremen
return New(t.identifier, newMeta, newLoc, t.fsF, t.cat), nil
}
+// SnapshotAsOf finds the snapshot that was current as of or right before the given timestamp.
+func (t Table) SnapshotAsOf(timestampMs int64, inclusive bool) *Snapshot {
+ entries := slices.Collect(t.metadata.SnapshotLogs())
+ for i := len(entries) - 1; i >= 0; i-- {
+ entry := entries[i]
+ if (inclusive && entry.TimestampMs <= timestampMs) || (!inclusive && entry.TimestampMs < timestampMs) {
+ return t.metadata.SnapshotByID(entry.SnapshotID)
+ }
+ }
+
+ return nil
+}
+
+// WithSnapshotAsOf creates a scan of the table as it existed at the given timestamp.
+// Note: Cannot be combined with WithSnapshotID option - time travel determines the snapshot.
+func (t Table) WithSnapshotAsOf(timestampMs int64, opts ...ScanOption) (*Scan, error) {
Review Comment:
yep, precisely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feate(table): Add time travel functionality [iceberg-go]
Posted by "dttung2905 (via GitHub)" <gi...@apache.org>.
dttung2905 commented on code in PR #548:
URL: https://github.com/apache/iceberg-go/pull/548#discussion_r2310353404
##########
table/table.go:
##########
@@ -227,6 +228,45 @@ func (t Table) doCommit(ctx context.Context, updates []Update, reqs []Requiremen
return New(t.identifier, newMeta, newLoc, t.fsF, t.cat), nil
}
+// SnapshotAsOf finds the snapshot that was current as of or right before the given timestamp.
+func (t Table) SnapshotAsOf(timestampMs int64, inclusive bool) *Snapshot {
+ entries := slices.Collect(t.metadata.SnapshotLogs())
+ for i := len(entries) - 1; i >= 0; i-- {
+ entry := entries[i]
+ if (inclusive && entry.TimestampMs <= timestampMs) || (!inclusive && entry.TimestampMs < timestampMs) {
+ return t.metadata.SnapshotByID(entry.SnapshotID)
+ }
+ }
+
+ return nil
+}
+
+// WithSnapshotAsOf creates a scan of the table as it existed at the given timestamp.
+// Note: Cannot be combined with WithSnapshotID option - time travel determines the snapshot.
+func (t Table) WithSnapshotAsOf(timestampMs int64, opts ...ScanOption) (*Scan, error) {
Review Comment:
Hi @zeroshade do you mean like this?
```golang
scan, err := table.Scan(
WithSnapshotAsOf(timestamp), // time travel ScanOption
WithLimit(100),
table.WithRowFilter("amount > 10")
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feate(table): Add time travel functionality [iceberg-go]
Posted by "dttung2905 (via GitHub)" <gi...@apache.org>.
dttung2905 commented on code in PR #548:
URL: https://github.com/apache/iceberg-go/pull/548#discussion_r2299160635
##########
table/table.go:
##########
@@ -227,6 +228,45 @@ func (t Table) doCommit(ctx context.Context, updates []Update, reqs []Requiremen
return New(t.identifier, newMeta, newLoc, t.fsF, t.cat), nil
}
+// TimeTravel finds the snapshot that was current as of or right before the given timestamp.
+func (t Table) TimeTravel(timestampMs int64, inclusive bool) *Snapshot {
Review Comment:
Thanks @zeroshade for the quick review. I have updated the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feate(table): Add time travel functionality [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #548:
URL: https://github.com/apache/iceberg-go/pull/548
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org