You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Chris Pettitt (JIRA)" <ji...@apache.org> on 2017/07/31 20:22:00 UTC

[jira] [Created] (BEAM-2703) KafkaIO: watermark outside the bounds of BoundedWindow

Chris Pettitt created BEAM-2703:
-----------------------------------

             Summary: KafkaIO: watermark outside the bounds of BoundedWindow
                 Key: BEAM-2703
                 URL: https://issues.apache.org/jira/browse/BEAM-2703
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-extensions
            Reporter: Chris Pettitt
            Assignee: Davor Bonaci


KafkaIO appears to use an incorrect lower bound for it's initial watermark with respect to BoundedWindow.TIMESTAMP_MIN_VALUE.

KafkaIO's initial watermark:
new Instant(Long.MIN_VALUE) -> -9223372036854775808

BoundedWindow.TIMESTAMP_MIN_VALUE:
new Instant(TimeUnit.MICROSECONDS.toMillis(Long.MIN_VALUE)) -> -9223372036854775

The difference is that the last three digits have been truncated due to the micro to millis conversion.

This difference can cause errors in runners that assert that the input watermark can never regress as KafkaIO gives a value below the lower bound when no messages have been received yet. For consistency it would probably be best for it to use BoundedWindow.TIMESTAMP_MIN_VALUE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)