[GitHub] [beam] lukecwik commented on pull request #15572: [BEAM-8218] WIP, PulsarIO, looking forward for initial feedback

[GitBox <gi...@apache.org>]

lukecwik commented on pull request #15572:
URL: https://github.com/apache/beam/pull/15572#issuecomment-949067175


   > Hi @lukecwik , I am working on the watermark estimator, and taking a look to KafkaIO, I see it has two types of watermark (MonotonicallyIncreasing and Manual), for first version of PulsarIO, I was consider only using MonotonicallyIncreasing, I want to know your input on this concern
   
   The purpose of the watermark tracking is to be able to fire certain triggers downstream of this transform instead of waiting till all the output of this transform is produced before being able to advance the watermark. In this case if the timestamp of the record is always equivalent to `currentOffset` then using `MonotonicallyIncreasing` makes sense. You'll also want to use `outputWithTimestamp(record, new Instant(currentOffset))` instead of the existing `output(record)` method.
   
   If the timestamp of the record is from some data associated within the record then it is dependent on the ordering the user provides and you'll want to provide some user some configuration options that allow you to convert a record to a timestamp and also to state how they want the watermark estimation to happen.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org