You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@paimon.apache.org by lec ssmi <sh...@gmail.com> on 2025/10/11 01:38:55 UTC
Clarify semantics of scan.snapshot-id in streaming vs batch (Paimon 1.2)
Hi Paimon community,
I’d like to confirm the intended semantics of *scan.snapshot-id* in Apache
Paimon *1.2* when reading with Flink.
What I see
- Table’s latest snapshot ID is *53*.
- I start a *streaming* query with:
SET 'execution.runtime-mode' = 'streaming';
SELECT * FROM my_table /*+ OPTIONS('scan.snapshot-id'='53') */;
- The job *emits records immediately*, even though there is *no snapshot
54* yet. After that initial output, it waits for new snapshots and
continues normally when new data arrives.
My understanding
- In streaming mode, when scan.snapshot-id is provided (and scan.mode
defaults to from-snapshot), the source *reads changes starting from that
snapshot* (i.e., includes the changes produced by snapshot S itself,
then S+1, S+2, …), and it *does not* first produce a full snapshot at
startup.
- In batch mode, using scan.snapshot-id = S should return the *full view
of snapshot S* only (no subsequent changes, no waiting).
Questions
1. *Streaming semantics:* Is it *by design* that from-snapshot is
*inclusive* of the starting snapshot’s changes (ΔS), i.e., it will
output ΔS even if there is no S+1 yet?
2. *Batch semantics:* Is it correct that batch + scan.snapshot-id = S
always returns *Full(S)* (and never ΔS)? Are there any exceptions
depending on table type?
3. *PK vs non-PK tables:* Should we expect any observable difference
here depending on whether the table is a primary-key table and/or the
changelog-producer is enabled (lookup, full-compaction, input)?
4. *Exclusive start recommendation:* If a user wants to *strictly start
from S+1* (i.e., exclude ΔS), is the recommended approach to:
- wait until S+1 exists and set scan.snapshot-id = S+1, or
- use incremental-between='S,S+1' for a bounded read?
5. *Docs wording:* If the above is the intended behavior, would it make
sense to emphasize the *inclusive* nature of from-snapshot in streaming
(vs. the *(start, end]* semantics of incremental-between) to help users
avoid confusion?
Environment:
- Paimon: 1.2
- Engine: Flink 1.19
- scan.mode: default (from-snapshot)
Thanks a lot for confirming the expected behavior .
Best regards
Re: Clarify semantics of scan.snapshot-id in streaming vs batch (Paimon 1.2)
Posted by Yunfeng Zhou <fl...@gmail.com>.
I agree with your understanding about the semantics of from-snapshot scan mode. I guess your Flink job might not be executing in streaming mode?
If all conditions are checked but the actual behavior is still different from the configuration document’s description, you can create an issue in Paimon repo to record this potential bug, with your detailed code to reproduce the unexpected behavior.
> 2025年10月11日 11:59,lec ssmi <sh...@gmail.com> 写道:
>
> What confuses me most is why, when I set the snapshot ID to the current latest, the job still produces records. From the documentation, my understanding is that in streaming mode this usage should not consume that snapshot itself, but rather use it as the starting point to consume incremental changes. This should also be one of the main differences between from-snapshot and from-snapshot-full, right?
>
>
> Yunfeng Zhou <flink.zhouyunfeng@gmail.com <ma...@gmail.com>> 于 2025年10月11日周六 11:15写道:
>> Hi lec ssmi,
>>
>> Most of your questions and understandings might be addressed by the document of the configuration `scan.mode`. You can find it here
>> https://paimon.apache.org/docs/master/maintenance/configurations/
>> It explains differences between modes like "from-snapshot” and “from-timestamp-full” (corresponds to your exclusive/inclusive converns) and the different behaviors between batch and streaming mode.
>>
>> The streaming read result does differ according to the changelog producer of the table. You can find the corresponding behaviors here
>> https://paimon.apache.org/docs/master/primary-key-table/changelog-producer/
>>
>> Best,
>> Yunfeng
>>
>>> 2025年10月11日 09:38,lec ssmi <shicheng31604@gmail.com <ma...@gmail.com>> 写道:
>>>
>>> Hi Paimon community,
>>>
>>> I’d like to confirm the intended semantics of scan.snapshot-id in Apache Paimon 1.2 when reading with Flink.
>>>
>>> What I see
>>>
>>> Table’s latest snapshot ID is 53.
>>> I start a streaming query with:
>>> SET 'execution.runtime-mode' = 'streaming';
>>> SELECT * FROM my_table /*+ OPTIONS('scan.snapshot-id'='53') */;
>>> The job emits records immediately, even though there is no snapshot 54 yet. After that initial output, it waits for new snapshots and continues normally when new data arrives.
>>> My understanding
>>>
>>> In streaming mode, when scan.snapshot-id is provided (and scan.mode defaults to from-snapshot), the source reads changes starting from that snapshot (i.e., includes the changes produced by snapshot S itself, then S+1, S+2, …), and it does not first produce a full snapshot at startup.
>>> In batch mode, using scan.snapshot-id = S should return the full view of snapshot S only (no subsequent changes, no waiting).
>>> Questions
>>>
>>> Streaming semantics: Is it by design that from-snapshot is inclusive of the starting snapshot’s changes (ΔS), i.e., it will output ΔS even if there is no S+1 yet?
>>> Batch semantics: Is it correct that batch + scan.snapshot-id = S always returns Full(S) (and never ΔS)? Are there any exceptions depending on table type?
>>> PK vs non-PK tables: Should we expect any observable difference here depending on whether the table is a primary-key table and/or the changelog-producer is enabled (lookup, full-compaction, input)?
>>> Exclusive start recommendation: If a user wants to strictly start from S+1 (i.e., exclude ΔS), is the recommended approach to:
>>> wait until S+1 exists and set scan.snapshot-id = S+1, or
>>> use incremental-between='S,S+1' for a bounded read?
>>> Docs wording: If the above is the intended behavior, would it make sense to emphasize the inclusive nature of from-snapshot in streaming (vs. the (start, end] semantics of incremental-between) to help users avoid confusion?
>>> Environment:
>>>
>>> Paimon: 1.2
>>> Engine: Flink 1.19
>>> scan.mode: default (from-snapshot)
>>> Thanks a lot for confirming the expected behavior .
>>>
>>> Best regards
>>>
>>
Re: Clarify semantics of scan.snapshot-id in streaming vs batch (Paimon 1.2)
Posted by lec ssmi <sh...@gmail.com>.
What confuses me most is why, when I set the snapshot ID to the current
latest, the job still produces records. From the documentation, my
understanding is that in streaming mode this usage should not consume that
snapshot itself, but rather use it as the starting point to consume
incremental changes. This should also be one of the main differences
between from-snapshot and from-snapshot-full, right?
Yunfeng Zhou <fl...@gmail.com> 于 2025年10月11日周六 11:15写道:
> Hi lec ssmi,
>
> Most of your questions and understandings might be addressed by the
> document of the configuration `scan.mode`. You can find it here
> https://paimon.apache.org/docs/master/maintenance/configurations/
> It explains differences between modes like "from-snapshot” and
> “from-timestamp-full” (corresponds to your exclusive/inclusive converns)
> and the different behaviors between batch and streaming mode.
>
> The streaming read result does differ according to the changelog producer
> of the table. You can find the corresponding behaviors here
> https://paimon.apache.org/docs/master/primary-key-table/changelog-producer/
>
> Best,
> Yunfeng
>
> 2025年10月11日 09:38,lec ssmi <sh...@gmail.com> 写道:
>
> Hi Paimon community,
>
> I’d like to confirm the intended semantics of *scan.snapshot-id* in
> Apache Paimon *1.2* when reading with Flink.
> What I see
>
> - Table’s latest snapshot ID is *53*.
> - I start a *streaming* query with:
>
> SET 'execution.runtime-mode' = 'streaming';
> SELECT * FROM my_table /*+ OPTIONS('scan.snapshot-id'='53') */;
>
> - The job *emits records immediately*, even though there is *no
> snapshot 54* yet. After that initial output, it waits for new
> snapshots and continues normally when new data arrives.
>
> My understanding
>
> - In streaming mode, when scan.snapshot-id is provided (and scan.mode
> defaults to from-snapshot), the source *reads changes starting from
> that snapshot* (i.e., includes the changes produced by snapshot S
> itself, then S+1, S+2, …), and it *does not* first produce a full
> snapshot at startup.
> - In batch mode, using scan.snapshot-id = S should return the *full
> view of snapshot S* only (no subsequent changes, no waiting).
>
> Questions
>
> 1. *Streaming semantics:* Is it *by design* that from-snapshot is
> *inclusive* of the starting snapshot’s changes (ΔS), i.e., it will
> output ΔS even if there is no S+1 yet?
> 2. *Batch semantics:* Is it correct that batch + scan.snapshot-id = S
> always returns *Full(S)* (and never ΔS)? Are there any exceptions
> depending on table type?
> 3. *PK vs non-PK tables:* Should we expect any observable difference
> here depending on whether the table is a primary-key table and/or the
> changelog-producer is enabled (lookup, full-compaction, input)?
> 4. *Exclusive start recommendation:* If a user wants to *strictly
> start from S+1* (i.e., exclude ΔS), is the recommended approach to:
> - wait until S+1 exists and set scan.snapshot-id = S+1, or
> - use incremental-between='S,S+1' for a bounded read?
> 5. *Docs wording:* If the above is the intended behavior, would it
> make sense to emphasize the *inclusive* nature of from-snapshot in
> streaming (vs. the *(start, end]* semantics of incremental-between) to
> help users avoid confusion?
>
> Environment:
>
> - Paimon: 1.2
> - Engine: Flink 1.19
> - scan.mode: default (from-snapshot)
>
> Thanks a lot for confirming the expected behavior .
>
> Best regards
>
>
>
Re: Clarify semantics of scan.snapshot-id in streaming vs batch (Paimon 1.2)
Posted by Yunfeng Zhou <fl...@gmail.com>.
Hi lec ssmi,
Most of your questions and understandings might be addressed by the document of the configuration `scan.mode`. You can find it here
https://paimon.apache.org/docs/master/maintenance/configurations/
It explains differences between modes like "from-snapshot” and “from-timestamp-full” (corresponds to your exclusive/inclusive converns) and the different behaviors between batch and streaming mode.
The streaming read result does differ according to the changelog producer of the table. You can find the corresponding behaviors here
https://paimon.apache.org/docs/master/primary-key-table/changelog-producer/
Best,
Yunfeng
> 2025年10月11日 09:38,lec ssmi <sh...@gmail.com> 写道:
>
> Hi Paimon community,
>
> I’d like to confirm the intended semantics of scan.snapshot-id in Apache Paimon 1.2 when reading with Flink.
>
> What I see
>
> Table’s latest snapshot ID is 53.
> I start a streaming query with:
> SET 'execution.runtime-mode' = 'streaming';
> SELECT * FROM my_table /*+ OPTIONS('scan.snapshot-id'='53') */;
> The job emits records immediately, even though there is no snapshot 54 yet. After that initial output, it waits for new snapshots and continues normally when new data arrives.
> My understanding
>
> In streaming mode, when scan.snapshot-id is provided (and scan.mode defaults to from-snapshot), the source reads changes starting from that snapshot (i.e., includes the changes produced by snapshot S itself, then S+1, S+2, …), and it does not first produce a full snapshot at startup.
> In batch mode, using scan.snapshot-id = S should return the full view of snapshot S only (no subsequent changes, no waiting).
> Questions
>
> Streaming semantics: Is it by design that from-snapshot is inclusive of the starting snapshot’s changes (ΔS), i.e., it will output ΔS even if there is no S+1 yet?
> Batch semantics: Is it correct that batch + scan.snapshot-id = S always returns Full(S) (and never ΔS)? Are there any exceptions depending on table type?
> PK vs non-PK tables: Should we expect any observable difference here depending on whether the table is a primary-key table and/or the changelog-producer is enabled (lookup, full-compaction, input)?
> Exclusive start recommendation: If a user wants to strictly start from S+1 (i.e., exclude ΔS), is the recommended approach to:
> wait until S+1 exists and set scan.snapshot-id = S+1, or
> use incremental-between='S,S+1' for a bounded read?
> Docs wording: If the above is the intended behavior, would it make sense to emphasize the inclusive nature of from-snapshot in streaming (vs. the (start, end] semantics of incremental-between) to help users avoid confusion?
> Environment:
>
> Paimon: 1.2
> Engine: Flink 1.19
> scan.mode: default (from-snapshot)
> Thanks a lot for confirming the expected behavior .
>
> Best regards
>