You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "mas-chen (via GitHub)" <gi...@apache.org> on 2023/06/27 21:40:36 UTC
[GitHub] [flink] mas-chen commented on a diff in pull request #22852: [FLINK-32420][connectors/common] Improve the watermark aggregation performance
mas-chen commented on code in PR #22852:
URL: https://github.com/apache/flink/pull/22852#discussion_r1244375639
##########
flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinator.java:
##########
@@ -631,8 +631,9 @@ private void ensureStarted() {
}
}
- private static class WatermarkAggregator<T> {
+ static class WatermarkAggregator<T> {
private final Map<T, Watermark> watermarks = new HashMap<>();
+ /** The aggregatedWatermark is the smallest watermark of all keys. */
Review Comment:
nit: I think the style is to use single line comments `//`. The multiline is reserved for signatures, classes, etc.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinator.java:
##########
@@ -642,7 +643,31 @@ private static class WatermarkAggregator<T> {
* Optional.empty()} otherwise.
*/
public Optional<Watermark> aggregate(T key, Watermark watermark) {
- watermarks.put(key, watermark);
+ Watermark oldWatermark = watermarks.put(key, watermark);
+ // Step (1): Update the latest watermark of current key as the aggregatedWatermark
+ // directly if it is less than the aggregatedWatermark.
+ if (watermark.getTimestamp() < aggregatedWatermark.getTimestamp()) {
+ aggregatedWatermark = watermark;
+ return Optional.of(aggregatedWatermark);
+ }
+
+ // Step(2): The aggWM won't change when these conditions are met, so return directly:
+ // case1. The latest WM of the current key isn't changed
+ // case2. When oldWatermark isn't null and is greater than aggWm, it means that aggWm
+ // comes from other keys. If new WM is greater than or equal to aggWm, then aggWm must
+ // not change.
+ // case3. When oldWatermark is null and {@link watermarks} has other keys, it means that
+ // aggWm comes from other keys. If new WM is greater than or equal to aggWm, then aggWm
+ // must not change.
+ // Note: Here's an implicit condition `watermark >= aggregatedWatermark` due to step(1),
+ // that's why it's not written in the code of case2 and case3.
+ if (watermark.equals(oldWatermark)
+ || (oldWatermark != null
+ && oldWatermark.getTimestamp() > aggregatedWatermark.getTimestamp())
+ || (oldWatermark == null && watermarks.size() > 1)) {
Review Comment:
nit
```suggestion
|| (oldWatermark == null && !watermarks.isEmpty())) {
```
##########
flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinator.java:
##########
@@ -642,7 +643,31 @@ private static class WatermarkAggregator<T> {
* Optional.empty()} otherwise.
*/
public Optional<Watermark> aggregate(T key, Watermark watermark) {
- watermarks.put(key, watermark);
+ Watermark oldWatermark = watermarks.put(key, watermark);
+ // Step (1): Update the latest watermark of current key as the aggregatedWatermark
+ // directly if it is less than the aggregatedWatermark.
+ if (watermark.getTimestamp() < aggregatedWatermark.getTimestamp()) {
+ aggregatedWatermark = watermark;
+ return Optional.of(aggregatedWatermark);
+ }
+
+ // Step(2): The aggWM won't change when these conditions are met, so return directly:
Review Comment:
When can `oldWatermark` be null since there is no remove() operation on `watermarks`?
##########
flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinator.java:
##########
@@ -631,8 +631,9 @@ private void ensureStarted() {
}
}
- private static class WatermarkAggregator<T> {
+ static class WatermarkAggregator<T> {
private final Map<T, Watermark> watermarks = new HashMap<>();
+ /** The aggregatedWatermark is the smallest watermark of all keys. */
Review Comment:
I was thinking that there's also further optimization on the data structure. The WatermarkAggregator only needs to return the min watermark and the set of tracked subtasks. So you could also use a TreeMap.
But inserting an element is now more costly and it's probably safe to assume watermark updates are more frequent than searching for the min watermark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org