Watermark-based trigger doesn't fire for 10+ minutes after message is received from Pub/Sub source

[Alex Mordkovich <am...@google.com>]

Hi Beam folks!

I'm running a simple Java Beam pipeline <https://pastebin.com/DY9X9ZVG> on
DirectRunner. The pipeline reads in messages from a Pub/Sub topic and
aggregates them into windows: by processing time and by event time. The
custom timestamp option isn't used, so the event time should be the
message's publish time.

What I'm observing is that the watermark-based trigger doesn't fire for 10+
minutes after the message is received by the pipeline:
https://pastebin.com/zgfDy5ej. I realize that PubsubIO.java's support for
watermarks is somewhat limited and relies on heuristics. However, looking
at how PubsubIO.Read computes watermarks
<https://github.com/apache/beam/blob/784d18b7ac89f87dd7fbf2861ee877f5b6070276/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L963>,
it looks like it will advance the watermark to now() if there are no
incoming messages for over a minute. So it doesn't look like the watermark
should get stuck. In fact, the comment here
<https://github.com/apache/beam/blob/784d18b7ac89f87dd7fbf2861ee877f5b6070276/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L94>
says that the watermark might get *ahead* of the true watermark, which
seems to be consistent with the code.

Is the behavior I'm observing actually expected currently? And if so, how
come?

Thanks in advance,
Alex