You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tzulitai <gi...@git.apache.org> on 2017/09/29 09:34:57 UTC
[GitHub] flink pull request #4747: [FLINK-7728] [DataStream] Flush StatusWatermarkVal...
GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/4747
[FLINK-7728] [DataStream] Flush StatusWatermarkValve once all inputs become idle
## What is the purpose of the change
This PR is based on #4738. Only the last three commits are relevant.
Prior to this PR, once all inputs of the `StatusWatermarkValve` becomes idle, we only emit the `StreamStatus.IDLE marker`, and check nothing else. This makes the watermark advancement behaviour inconsistent in the case that all inputs become idle, depending on the order that they become idle.
This PR fixes this by "flushing" the max watermark across all channels once all inputs become idle. At a high-level, what this means for downstream operators is that all inputs have become idle and will temporarily cease to advance their watermarks, so they can safely advance their event time to whatever the largest watermark is.
This PR also includes changes to `StatusWatermarkValveTest` to cover the above mentioned case, as well as reworking the unit tests to be more fine-grained with small enough test case scope, so that we have a better overview of what cases are covered.
## Brief change log
- d11d98d preliminary cleanup of `StatusWatermarkValveTest#BufferedValveOutputHandler`. Previous implementation was overly complex and could not provide test behaviors that we required.
- 0e386da main change for this PR. Includes a new test in `StatusWatermarkValve` to cover the missing case. That test does not pass without this change.
- b977c18 refactor unit tests to be more fine-grained scoped.
## Verifying this change
`StatusWatermarkValveTest`, `OneInputStreamTaskTest`, `TwoInputStreamTaskTest` should be able to verify this change.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): **YES**
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? n/a
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-7728
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4747.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4747
----
commit a9ce90aaee99374513941270b804de2f5564c47e
Author: Tzu-Li (Gordon) Tai <tz...@apache.org>
Date: 2017-09-27T18:05:21Z
[FLINK-7721] [DataStream] Only emit new min watermark iff it was aggregated from watermark-aligned inputs
Prior to this commit, In the calculation of the new min watermark in
StatusWatermarkValve#findAndOutputNewMinWatermarkAcrossAlignedChannels(),
there is no verification that the calculated new min watermark
really is aggregated from some aligned channel.
In the corner case where all input channels are currently not aligned
but actually some are active, we would then incorrectly determine that
the final aggregation is Long.MAX_VALUE and emit that.
This commit fixes this by only emitting the aggregated watermark iff it was
really calculated from some aligned input channel (as well as the
already existing constraint that it needs to be larger than the last
emitted watermark). This change should also safely cover the case that a
Long.MAX_VALUE was genuinely aggregated from the input channels.
commit d11d98dda4723760e7d9fb3b9680a1e9daca3705
Author: Tzu-Li (Gordon) Tai <tz...@apache.org>
Date: 2017-09-28T12:11:22Z
[FLINK-7728] [DataStream] Simplify BufferedValveOutputHandler used in StatusWatermarkValveTest
The previous implementation was overly complicated. Having separate
buffers for the StreamStatus and Watermarks is not required for our
tests. Also, that design doesn't allow checking the order StreamStatus /
Watermarks are emitted from a single input to the valve.
This commit reworks it by buffering both StreamStatus and Watermarks in
a shared queue.
commit 0e386dab2fc19df78276ce203ed8f38792145759
Author: Tzu-Li (Gordon) Tai <tz...@apache.org>
Date: 2017-09-28T14:49:22Z
[FLINK-7728] [DataStream] Flush max watermark across all inputs once all become idle
Prior to this commit, once all inputs of the StatusWatermarkValve
becomes idle, we only emit the StreamStatus.IDLE marker, and check
nothing else. This makes the watermark advancement behaviour
inconsistent in the case that all inputs become idle depending on the
order that they become idle.
This commit fixes this by "flushing" the max watermark across all
channels once all inputs become idle. At a high-level, what this means
for downstream operators is that all inputs have become idle and will
temporariliy cease to advance their watermarks, so they can safely
advance their event time to whatever the largest watermark is.
commit b977c18c802fcb0060b47d7079bf214da7e764bb
Author: Tzu-Li (Gordon) Tai <tz...@apache.org>
Date: 2017-09-29T09:16:22Z
[FLINK-7728] [DataStream] Make StatusWatermarkValve unit tests more fine-grained
Previously, the unit tests in StatusWatermarkValveTest were too
cluttered and testing too many behaviours in a single test. This makes
it hard to have a good overview of what test cases are covered.
This commit is a rework of the previous tests, making them more
fine-grained so that the scope of each test is small enough. All
previously tested behaviours are still covered.
----
---
[GitHub] flink issue #4747: [FLINK-7728] [DataStream] Flush StatusWatermarkValve once...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/4747
This looks very good, the test is also much cleaner and understandable now. 👌
One thing, in the refactoring the bit of testing code that verifies the fix for FLINK-7721 was dropped (I checked this by commenting out the fix in the code). I'll re-add a test for that and merge.
---
[GitHub] flink issue #4747: [FLINK-7728] [DataStream] Flush StatusWatermarkValve once...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4747
Thanks for merging @aljoscha. Closing this ..
---
[GitHub] flink issue #4747: [FLINK-7728] [DataStream] Flush StatusWatermarkValve once...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/4747
Merged. 👌 Could you please also close this PR?
---
[GitHub] flink pull request #4747: [FLINK-7728] [DataStream] Flush StatusWatermarkVal...
Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai closed the pull request at:
https://github.com/apache/flink/pull/4747
---