You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "wwj6591812 (via GitHub)" <gi...@apache.org> on 2026/08/10 07:29:26 UTC
[PR] [core] Support latest snapshot delta scan in batch [paimon]
wwj6591812 opened a new pull request, #9142:
URL: https://github.com/apache/paimon/pull/9142
### Purpose
Batch pipelines sometimes need to process only the newest committed snapshot, but currently must first query its ID and construct `incremental-between` boundaries.
This adds `scan.mode = 'latest-delta'` for batch sources. At planning time it selects the literal latest snapshot `N` and reuses the existing DELTA scanner for `(N - 1, N]`.
It deliberately does not search backwards for an APPEND snapshot. Therefore, if the latest snapshot is COMPACT or OVERWRITE, the query returns no records instead of treating rewritten files as new logical data. Streaming use is rejected, and boundary or time-travel companion options are rejected rather than ignored.
This is a convenience mode, not a persistent consumer cursor. Workflows that must consume multiple snapshots reliably should continue to use explicit `incremental-between` boundaries or tags.
```sql
SELECT *
FROM t /*+ OPTIONS('scan.mode' = 'latest-delta') */;
```
### Tests
- Added core coverage for latest APPEND, latest COMPACT, empty table, streaming rejection, level-0 preservation, incompatible options, and postpone-merge rejection.
- Added a Flink SQL integration test proving two appends return only the latest snapshot.
- Added Flink catalog option coverage for batch and streaming modes.
- Regenerated configuration docs with the official docs profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support latest snapshot delta scan in batch [paimon]
Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #9142:
URL: https://github.com/apache/paimon/pull/9142#issuecomment-5265839171
@JingsongLi hi,please cc, thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support latest snapshot delta scan in batch [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9142:
URL: https://github.com/apache/paimon/pull/9142#discussion_r3763395669
##########
docs/docs/flink/sql-query.mdx:
##########
@@ -110,6 +110,21 @@ If you want see `DELETE` records, you can use audit_log table:
SELECT * FROM t$audit_log /*+ OPTIONS('incremental-between' = '12,20') */;
```
+### Batch Latest Delta
Review Comment:
Do not add doc for this internal usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support latest snapshot delta scan in batch [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9142:
URL: https://github.com/apache/paimon/pull/9142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support latest snapshot delta scan in batch [paimon]
Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #9142:
URL: https://github.com/apache/paimon/pull/9142#issuecomment-5275592498
@JingsongLi
hi,please cc, thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support latest snapshot delta scan in batch [paimon]
Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #9142:
URL: https://github.com/apache/paimon/pull/9142#discussion_r3764008539
##########
docs/docs/flink/sql-query.mdx:
##########
@@ -110,6 +110,21 @@ If you want see `DELETE` records, you can use audit_log table:
SELECT * FROM t$audit_log /*+ OPTIONS('incremental-between' = '12,20') */;
```
+### Batch Latest Delta
Review Comment:
Thanks. Removed the user-facing Batch Latest Delta section from sql-query.mdx in b8f2fb5b8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org