You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/10/20 19:07:46 UTC

[GitHub] [beam] damccorm opened a new issue, #23763: [Bug]: Periodic sequence never updates the watermark

damccorm opened a new issue, #23763:
URL: https://github.com/apache/beam/issues/23763

   ### What happened?
   
   Right now, if you have a streaming job with periodic impulse as your source, the watermark will not get updated until the periodic impulse completes.
   
   This means that if you have something like:
   
   ```
   pipeline
   | PeriodicImpulse(time.now(), time.now()+1000000, 1)
   | beam.Map(do_stuff)
   | beam.WindowInto(window.FixedWindows(2))
   | beam.combiners.Count.PerElement()
   | beam.Map(log_output)
   ```
   
   the pipeline will hang indefinitely and never log anything. If we properly emitted watermarks, it would log output every 2 seconds.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: sdk-py-core


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on issue #23763: [Bug]: Periodic sequence never updates the watermark

Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #23763:
URL: https://github.com/apache/beam/issues/23763#issuecomment-1286042589

   Wondering if this is the cause of #22776


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on issue #23763: [Bug]: Periodic sequence never updates the watermark

Posted by GitBox <gi...@apache.org>.
damccorm commented on issue #23763:
URL: https://github.com/apache/beam/issues/23763#issuecomment-1286061016

   It doesn't seem to be; I have a fix for the watermark issue, but it doesn't seem to make drain work.
   
   I think that this is probably because we use a bounded restriction but don't have a truncate function defined (see https://github.com/apache/beam/blob/f0e003bbaa074b2096f7c29aa8c133c0fd318587/sdks/python/apache_beam/io/iobase.py#L1382)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm closed issue #23763: [Bug]: Periodic sequence never updates the watermark

Posted by GitBox <gi...@apache.org>.
damccorm closed issue #23763: [Bug]: Periodic sequence never updates the watermark
URL: https://github.com/apache/beam/issues/23763


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on issue #23763: [Bug]: Periodic sequence never updates the watermark

Posted by GitBox <gi...@apache.org>.
damccorm commented on issue #23763:
URL: https://github.com/apache/beam/issues/23763#issuecomment-1286062767

   Scratch that (ignore my previous now deleted comment). This seems to fix the problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on issue #23763: [Bug]: Periodic sequence never updates the watermark

Posted by GitBox <gi...@apache.org>.
damccorm commented on issue #23763:
URL: https://github.com/apache/beam/issues/23763#issuecomment-1286082299

   Scratch that - my original analysis was right and we need to explicitly truncate. The drain I thought I saw was actually just my test framework cancelling the job 😳 Explicitly truncating works perfectly though, I'll put up a PR for that shortly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org