[I] [Feature] Add a global monotonically increasing timestamp service (TSO) for incremental computation in Doris [doris]

["AntiTopQuark (via GitHub)" <gi...@apache.org>]

AntiTopQuark opened a new issue, #61198:
URL: https://github.com/apache/doris/issues/61198

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Description
   
   Doris is introducing incremental computation capabilities in #57921, including table stream, dynamic table, incremental runtime, and optimizer support. To support these components consistently across the cluster, Doris needs a global timestamp service that can generate monotonically increasing, globally ordered timestamps.
   
   This TSO service can be used as a unified progress/version marker for incremental batches, CDC events, stream operators, and incremental materialized view refresh. It helps different FE/BE nodes reason about ordering and progress in a deterministic way, instead of relying only on local machine clocks.
   
   A TSO-like design has been widely used in distributed systems, where a centralized timestamp oracle allocates globally ordered timestamps using a physical time part plus a logical counter part.
   
   ### Use case
   
   - Provide globally ordered versions for table stream / Binlog
   - Mark incremental MV refresh boundaries
   - Support deterministic ordering for distributed stream join / aggregation
   
   ### Related issues
   
   This feature mainly solves the lack of a reliable cluster-wide ordering mechanism in distributed incremental computation. Local clocks are not sufficient to establish a strict global order across nodes. A TSO service provides a single monotonic timeline for change processing, operator progress, state advancement, and incremental refresh.
   
   It is suitable for table stream, dynamic table, incremental runtime, asynchronous MV refresh, distributed stream processing, and other internal components introduced by #57921.
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

Re: [I] [Feature] Add a global monotonically increasing timestamp service (TSO) for incremental computation in Doris [doris]

["morningman (via GitHub)" <gi...@apache.org>]

morningman closed issue #61198: [Feature] Add a global monotonically increasing timestamp service (TSO) for incremental computation in Doris
URL: https://github.com/apache/doris/issues/61198


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org