[GitHub] [iceberg] kbendick commented on pull request #3039: Introduce spark3 option to read stream from a timestamp

[GitBox <gi...@apache.org>]

kbendick commented on pull request #3039:
URL: https://github.com/apache/iceberg/pull/3039#issuecomment-914800861


   > > Can we please use `as-of-timestamp` instead of introducing a new configuration option? If users need to be strictly greater than some time or strictly less than some time, I believe they can use `as-of-timestamp` and simply add one to the milliseconds as we don’t store snapshot timestamps at a lower granularity than milliseconds iirc (will double check the spec but I’m almost certain).
   > > We need `as-of-timestamp` to be consistent with the batch API and Flink streaming APIs anyway and it already exists.
   > > If there's a motivating reason that we _can't_ use `as-of-timestamp`, I'd love to hear it. Very open to the existence of a possibility that I didn't consider which would prevent the use of `as-of-timestamp`. 🙂
   > 
   > @kbendick - the existing flag - `as-of-timestamp` is intended to be use for time travel scenarios. Using this flag - users would read full table data.
   > whereas - the new flag `stream-from-timestamp` is for streaming reads. In this case - the flag need to explicitly express that - user is interested to read data that is added to the table - **from** this timestamp - **NOT** the existing data - but the new data that is added to that table starting **FROM** this timestamp...
   > 
   > Did this rationale make sense!?
   
   Hi @SreeramGarlapati, yeah that distinction makes sense. Thanks for clarifying. Can we put something similar to your last line (`We need this config as opposed to `as-of-timestamp` as we only want to get data written after the specified timestamp`) as a comment?
   
   It's clear now that it's said, but it was admittedly a little confusing for me prior to that. I'll leave any clarification up to you two. Thanks for the info.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org