You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "1996fanrui (via GitHub)" <gi...@apache.org> on 2023/06/30 17:20:04 UTC

[GitHub] [flink] 1996fanrui opened a new pull request, #22921: [FLINK-32506][connectors/common] Add the benchmark for watermark aggregation when enabling the watermark alignment

1996fanrui opened a new pull request, #22921:
URL: https://github.com/apache/flink/pull/22921

   ## What is the purpose of the change
   
   [FLINK-32420](https://issues.apache.org/jira/browse/FLINK-32420) is improving the watermark aggregation performance.
   
   We want to add a benchmark for it first, and then we can see the official performance change.
   
   ## Brief change log
   
   Add the SourceCoordinatorAlignmentBenchmark.
   
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
     - *Added the  SourceCoordinatorAlignmentBenchmarkTest*
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): ( no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no)
     - The serializers: (no)
     - The runtime per-record code paths (performance sensitive): (no)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (no)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] 1996fanrui commented on a diff in pull request #22921: [FLINK-32506][connectors/common] Add the benchmark for watermark aggregation when enabling the watermark alignment

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #22921:
URL: https://github.com/apache/flink/pull/22921#discussion_r1251126912


##########
flink-runtime/src/test/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorAlignmentBenchmark.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.source.coordinator;
+
+import org.apache.flink.api.common.eventtime.WatermarkAlignmentParams;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.mocks.MockSource;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplit;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.apache.flink.runtime.source.event.ReportedWatermarkEvent;
+
+import java.util.Random;
+
+import static org.apache.flink.runtime.source.coordinator.CoordinatorTestUtils.waitForCoordinatorToProcessActions;
+
+/** The benchmark of watermark alignment. */
+public class SourceCoordinatorAlignmentBenchmark {
+
+    private static final OperatorID OPERATOR_ID = new OperatorID(1234L, 5678L);
+
+    private SourceCoordinator<?, ?> sourceCoordinator;
+
+    private int numSubtasks;
+
+    private long second;
+
+    private long[] randomMilliseconds;
+
+    public SourceCoordinatorAlignmentBenchmark() {}
+
+    public void setup(int numSubtasks) throws Exception {
+        SourceCoordinatorProvider<MockSourceSplit> provider =
+                new SourceCoordinatorProvider<>(
+                        "SourceCoordinatorProviderTest",
+                        OPERATOR_ID,
+                        new MockSource(Boundedness.BOUNDED, 2),
+                        1,
+                        new WatermarkAlignmentParams(1000L, "group1", Long.MAX_VALUE),
+                        null);
+        this.sourceCoordinator =
+                (SourceCoordinator<?, ?>)
+                        provider.getCoordinator(
+                                new MockOperatorCoordinatorContext(OPERATOR_ID, numSubtasks));
+        this.sourceCoordinator.start();
+        this.numSubtasks = numSubtasks;
+        this.second = 0;
+        this.randomMilliseconds = generateRandomMilliseconds(numSubtasks);
+
+        // Initialize the watermark for all subtasks.
+        sendReportedWatermarkToAllSubtasks();
+    }
+
+    public void teardown() throws Exception {
+        sourceCoordinator.close();
+    }
+
+    public void sendReportedWatermarkToAllSubtasks() {
+        for (int subtaskIndex = 0; subtaskIndex < numSubtasks; subtaskIndex++) {
+            sourceCoordinator.handleEventFromOperator(
+                    subtaskIndex,
+                    0,
+                    new ReportedWatermarkEvent(second + randomMilliseconds[subtaskIndex]));
+        }
+        waitForCoordinatorToProcessActions(sourceCoordinator.getContext());
+        second += 1000;

Review Comment:
   Thanks for the quick feedback.
   
   Sounds make sense, updated to `100_000`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] 1996fanrui commented on pull request #22921: [FLINK-32506][connectors/common] Add the benchmark for watermark aggregation when enabling the watermark alignment

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #22921:
URL: https://github.com/apache/flink/pull/22921#issuecomment-1615396226

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] pnowojski commented on a diff in pull request #22921: [FLINK-32506][connectors/common] Add the benchmark for watermark aggregation when enabling the watermark alignment

Posted by "pnowojski (via GitHub)" <gi...@apache.org>.
pnowojski commented on code in PR #22921:
URL: https://github.com/apache/flink/pull/22921#discussion_r1250900302


##########
flink-runtime/src/test/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorAlignmentBenchmark.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.source.coordinator;
+
+import org.apache.flink.api.common.eventtime.WatermarkAlignmentParams;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.mocks.MockSource;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplit;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.apache.flink.runtime.source.event.ReportedWatermarkEvent;
+
+import javax.annotation.Nonnull;
+
+import java.util.Random;
+
+import static org.apache.flink.runtime.source.coordinator.CoordinatorTestUtils.waitForCoordinatorToProcessActions;
+
+/** The benchmark of watermark alignment. */
+public class SourceCoordinatorAlignmentBenchmark {
+
+    /** The TimestampType determines how the timestamp or watermark is generated. */
+    public enum TimestampType {
+        MONOTONE_INCREASING,
+        RANDOMIZE_MILLISECONDS
+    }
+
+    private static final OperatorID OPERATOR_ID = new OperatorID(1234L, 5678L);
+    private static final Random RANDOM = new Random();
+
+    private SourceCoordinator<?, ?> sourceCoordinator;
+    private int numSubtasks;
+    private TimestampType timestampType;
+
+    public SourceCoordinatorAlignmentBenchmark() {}
+
+    public void setup(int numSubtasks, @Nonnull TimestampType timestampType) throws Exception {
+        SourceCoordinatorProvider<MockSourceSplit> provider =
+                new SourceCoordinatorProvider<>(
+                        "SourceCoordinatorProviderTest",
+                        OPERATOR_ID,
+                        new MockSource(Boundedness.BOUNDED, 2),
+                        1,
+                        new WatermarkAlignmentParams(1000L, "group1", Long.MAX_VALUE),
+                        null);
+        this.sourceCoordinator =
+                (SourceCoordinator<?, ?>)
+                        provider.getCoordinator(
+                                new MockOperatorCoordinatorContext(OPERATOR_ID, numSubtasks));
+        this.sourceCoordinator.start();
+        this.numSubtasks = numSubtasks;
+        this.timestampType = timestampType;
+
+        // Initialize the watermark for all subtasks.
+        sendReportedWatermarkToAllSubtasks();
+    }
+
+    public void teardown() throws Exception {
+        sourceCoordinator.close();
+    }
+
+    public void sendReportedWatermarkToAllSubtasks() {

Review Comment:
   can you verify with some code profiler, that when calling this method repeatedly for a minute or so, what are you actually benchmarking? Where does the CPU spend most of its time? If those are the exact places that you would expect in production code, then all is good. However it shouldn't be things like `RANDOM.nextInt()` or `System.currentTimeMillis()` or other unrelated call.



##########
flink-runtime/src/test/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorAlignmentBenchmark.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.source.coordinator;
+
+import org.apache.flink.api.common.eventtime.WatermarkAlignmentParams;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.mocks.MockSource;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplit;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.apache.flink.runtime.source.event.ReportedWatermarkEvent;
+
+import javax.annotation.Nonnull;
+
+import java.util.Random;
+
+import static org.apache.flink.runtime.source.coordinator.CoordinatorTestUtils.waitForCoordinatorToProcessActions;
+
+/** The benchmark of watermark alignment. */
+public class SourceCoordinatorAlignmentBenchmark {
+
+    /** The TimestampType determines how the timestamp or watermark is generated. */
+    public enum TimestampType {
+        MONOTONE_INCREASING,
+        RANDOMIZE_MILLISECONDS
+    }
+
+    private static final OperatorID OPERATOR_ID = new OperatorID(1234L, 5678L);
+    private static final Random RANDOM = new Random();
+
+    private SourceCoordinator<?, ?> sourceCoordinator;
+    private int numSubtasks;
+    private TimestampType timestampType;
+
+    public SourceCoordinatorAlignmentBenchmark() {}
+
+    public void setup(int numSubtasks, @Nonnull TimestampType timestampType) throws Exception {
+        SourceCoordinatorProvider<MockSourceSplit> provider =
+                new SourceCoordinatorProvider<>(
+                        "SourceCoordinatorProviderTest",
+                        OPERATOR_ID,
+                        new MockSource(Boundedness.BOUNDED, 2),
+                        1,
+                        new WatermarkAlignmentParams(1000L, "group1", Long.MAX_VALUE),
+                        null);
+        this.sourceCoordinator =
+                (SourceCoordinator<?, ?>)
+                        provider.getCoordinator(
+                                new MockOperatorCoordinatorContext(OPERATOR_ID, numSubtasks));
+        this.sourceCoordinator.start();
+        this.numSubtasks = numSubtasks;
+        this.timestampType = timestampType;
+
+        // Initialize the watermark for all subtasks.
+        sendReportedWatermarkToAllSubtasks();
+    }
+
+    public void teardown() throws Exception {
+        sourceCoordinator.close();
+    }
+
+    public void sendReportedWatermarkToAllSubtasks() {
+        for (int subtaskIndex = 0; subtaskIndex < numSubtasks; subtaskIndex++) {
+            sourceCoordinator.handleEventFromOperator(
+                    subtaskIndex, 0, new ReportedWatermarkEvent(generateTimestamp()));
+        }
+        waitForCoordinatorToProcessActions(sourceCoordinator.getContext());
+    }
+
+    private long generateTimestamp() {
+        switch (timestampType) {
+            case MONOTONE_INCREASING:
+                return System.currentTimeMillis();
+            case RANDOMIZE_MILLISECONDS:
+                return System.currentTimeMillis() / 1000 * 1000 + RANDOM.nextInt(1000);

Review Comment:
   Don't you actually benchmark here `RANDOM.nextInt()` and `System.currentTimeMillis()` instead of the `SourceCoordinator`? 🤔 `Random` is a very heavy class with lot's of overheads. `ThreadLocalRandom` is better, but there is no good replacement of `System.currentTimeMillis`, which is also quite heavy (syscall). 
   
   Usually the solution is have some pre-generated small array of timestamps in the `setup` method, that's later used in the benchmark. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] 1996fanrui merged pull request #22921: [FLINK-32506][connectors/common] Add the benchmark for watermark aggregation when enabling the watermark alignment

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui merged PR #22921:
URL: https://github.com/apache/flink/pull/22921


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] flinkbot commented on pull request #22921: [FLINK-32506][connectors/common] Add the benchmark for watermark aggregation when enabling the watermark alignment

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #22921:
URL: https://github.com/apache/flink/pull/22921#issuecomment-1614960838

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "73edc3a766af1d716a28c63b6729042ac94eb12d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "73edc3a766af1d716a28c63b6729042ac94eb12d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 73edc3a766af1d716a28c63b6729042ac94eb12d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] pnowojski commented on a diff in pull request #22921: [FLINK-32506][connectors/common] Add the benchmark for watermark aggregation when enabling the watermark alignment

Posted by "pnowojski (via GitHub)" <gi...@apache.org>.
pnowojski commented on code in PR #22921:
URL: https://github.com/apache/flink/pull/22921#discussion_r1251102882


##########
flink-runtime/src/test/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorAlignmentBenchmark.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.source.coordinator;
+
+import org.apache.flink.api.common.eventtime.WatermarkAlignmentParams;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.mocks.MockSource;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplit;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.apache.flink.runtime.source.event.ReportedWatermarkEvent;
+
+import java.util.Random;
+
+import static org.apache.flink.runtime.source.coordinator.CoordinatorTestUtils.waitForCoordinatorToProcessActions;
+
+/** The benchmark of watermark alignment. */
+public class SourceCoordinatorAlignmentBenchmark {
+
+    private static final OperatorID OPERATOR_ID = new OperatorID(1234L, 5678L);
+
+    private SourceCoordinator<?, ?> sourceCoordinator;
+
+    private int numSubtasks;
+
+    private long second;
+
+    private long[] randomMilliseconds;
+
+    public SourceCoordinatorAlignmentBenchmark() {}
+
+    public void setup(int numSubtasks) throws Exception {
+        SourceCoordinatorProvider<MockSourceSplit> provider =
+                new SourceCoordinatorProvider<>(
+                        "SourceCoordinatorProviderTest",
+                        OPERATOR_ID,
+                        new MockSource(Boundedness.BOUNDED, 2),
+                        1,
+                        new WatermarkAlignmentParams(1000L, "group1", Long.MAX_VALUE),
+                        null);
+        this.sourceCoordinator =
+                (SourceCoordinator<?, ?>)
+                        provider.getCoordinator(
+                                new MockOperatorCoordinatorContext(OPERATOR_ID, numSubtasks));
+        this.sourceCoordinator.start();
+        this.numSubtasks = numSubtasks;
+        this.second = 0;
+        this.randomMilliseconds = generateRandomMilliseconds(numSubtasks);
+
+        // Initialize the watermark for all subtasks.
+        sendReportedWatermarkToAllSubtasks();
+    }
+
+    public void teardown() throws Exception {
+        sourceCoordinator.close();
+    }
+
+    public void sendReportedWatermarkToAllSubtasks() {
+        for (int subtaskIndex = 0; subtaskIndex < numSubtasks; subtaskIndex++) {
+            sourceCoordinator.handleEventFromOperator(
+                    subtaskIndex,
+                    0,
+                    new ReportedWatermarkEvent(second + randomMilliseconds[subtaskIndex]));
+        }
+        waitForCoordinatorToProcessActions(sourceCoordinator.getContext());
+        second += 1000;

Review Comment:
   nit: What would happen if single `sendReportedWatermarkToAllSubtasks` takes longer then `1000ms`? Would it cause some problems? If so, maybe change this to `10_000` or even `100_000`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] 1996fanrui commented on a diff in pull request #22921: [FLINK-32506][connectors/common] Add the benchmark for watermark aggregation when enabling the watermark alignment

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #22921:
URL: https://github.com/apache/flink/pull/22921#discussion_r1251074496


##########
flink-runtime/src/test/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorAlignmentBenchmark.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.source.coordinator;
+
+import org.apache.flink.api.common.eventtime.WatermarkAlignmentParams;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.mocks.MockSource;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplit;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.apache.flink.runtime.source.event.ReportedWatermarkEvent;
+
+import javax.annotation.Nonnull;
+
+import java.util.Random;
+
+import static org.apache.flink.runtime.source.coordinator.CoordinatorTestUtils.waitForCoordinatorToProcessActions;
+
+/** The benchmark of watermark alignment. */
+public class SourceCoordinatorAlignmentBenchmark {
+
+    /** The TimestampType determines how the timestamp or watermark is generated. */
+    public enum TimestampType {
+        MONOTONE_INCREASING,
+        RANDOMIZE_MILLISECONDS
+    }
+
+    private static final OperatorID OPERATOR_ID = new OperatorID(1234L, 5678L);
+    private static final Random RANDOM = new Random();
+
+    private SourceCoordinator<?, ?> sourceCoordinator;
+    private int numSubtasks;
+    private TimestampType timestampType;
+
+    public SourceCoordinatorAlignmentBenchmark() {}
+
+    public void setup(int numSubtasks, @Nonnull TimestampType timestampType) throws Exception {
+        SourceCoordinatorProvider<MockSourceSplit> provider =
+                new SourceCoordinatorProvider<>(
+                        "SourceCoordinatorProviderTest",
+                        OPERATOR_ID,
+                        new MockSource(Boundedness.BOUNDED, 2),
+                        1,
+                        new WatermarkAlignmentParams(1000L, "group1", Long.MAX_VALUE),
+                        null);
+        this.sourceCoordinator =
+                (SourceCoordinator<?, ?>)
+                        provider.getCoordinator(
+                                new MockOperatorCoordinatorContext(OPERATOR_ID, numSubtasks));
+        this.sourceCoordinator.start();
+        this.numSubtasks = numSubtasks;
+        this.timestampType = timestampType;
+
+        // Initialize the watermark for all subtasks.
+        sendReportedWatermarkToAllSubtasks();
+    }
+
+    public void teardown() throws Exception {
+        sourceCoordinator.close();
+    }
+
+    public void sendReportedWatermarkToAllSubtasks() {
+        for (int subtaskIndex = 0; subtaskIndex < numSubtasks; subtaskIndex++) {
+            sourceCoordinator.handleEventFromOperator(
+                    subtaskIndex, 0, new ReportedWatermarkEvent(generateTimestamp()));
+        }
+        waitForCoordinatorToProcessActions(sourceCoordinator.getContext());
+    }
+
+    private long generateTimestamp() {
+        switch (timestampType) {
+            case MONOTONE_INCREASING:
+                return System.currentTimeMillis();
+            case RANDOMIZE_MILLISECONDS:
+                return System.currentTimeMillis() / 1000 * 1000 + RANDOM.nextInt(1000);

Review Comment:
   Hi @pnowojski , thanks a lot for your review and great suggestion.
   
   I have updated this PR and https://github.com/apache/flink-benchmarks/pull/77 based on your suggestion:
   
   - Removed the `random` and `System.currentTimeMillis` during the benchmark
   - Removed the `TimestampType`, just using the second + random_ms



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] 1996fanrui commented on pull request #22921: [FLINK-32506][connectors/common] Add the benchmark for watermark aggregation when enabling the watermark alignment

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #22921:
URL: https://github.com/apache/flink/pull/22921#issuecomment-1619503606

   Thanks a lot @RocMarshal and @pnowojski 's review and the great suggestion, merging~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] pnowojski commented on a diff in pull request #22921: [FLINK-32506][connectors/common] Add the benchmark for watermark aggregation when enabling the watermark alignment

Posted by "pnowojski (via GitHub)" <gi...@apache.org>.
pnowojski commented on code in PR #22921:
URL: https://github.com/apache/flink/pull/22921#discussion_r1250896647


##########
flink-runtime/src/test/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorAlignmentBenchmark.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.source.coordinator;
+
+import org.apache.flink.api.common.eventtime.WatermarkAlignmentParams;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.mocks.MockSource;
+import org.apache.flink.api.connector.source.mocks.MockSourceSplit;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.operators.coordination.MockOperatorCoordinatorContext;
+import org.apache.flink.runtime.source.event.ReportedWatermarkEvent;
+
+import javax.annotation.Nonnull;
+
+import java.util.Random;
+
+import static org.apache.flink.runtime.source.coordinator.CoordinatorTestUtils.waitForCoordinatorToProcessActions;
+
+/** The benchmark of watermark alignment. */
+public class SourceCoordinatorAlignmentBenchmark {
+
+    /** The TimestampType determines how the timestamp or watermark is generated. */
+    public enum TimestampType {
+        MONOTONE_INCREASING,
+        RANDOMIZE_MILLISECONDS
+    }
+
+    private static final OperatorID OPERATOR_ID = new OperatorID(1234L, 5678L);
+    private static final Random RANDOM = new Random();
+
+    private SourceCoordinator<?, ?> sourceCoordinator;
+    private int numSubtasks;
+    private TimestampType timestampType;
+
+    public SourceCoordinatorAlignmentBenchmark() {}
+
+    public void setup(int numSubtasks, @Nonnull TimestampType timestampType) throws Exception {
+        SourceCoordinatorProvider<MockSourceSplit> provider =
+                new SourceCoordinatorProvider<>(
+                        "SourceCoordinatorProviderTest",
+                        OPERATOR_ID,
+                        new MockSource(Boundedness.BOUNDED, 2),
+                        1,
+                        new WatermarkAlignmentParams(1000L, "group1", Long.MAX_VALUE),
+                        null);
+        this.sourceCoordinator =
+                (SourceCoordinator<?, ?>)
+                        provider.getCoordinator(
+                                new MockOperatorCoordinatorContext(OPERATOR_ID, numSubtasks));
+        this.sourceCoordinator.start();
+        this.numSubtasks = numSubtasks;
+        this.timestampType = timestampType;
+
+        // Initialize the watermark for all subtasks.
+        sendReportedWatermarkToAllSubtasks();
+    }
+
+    public void teardown() throws Exception {
+        sourceCoordinator.close();
+    }
+
+    public void sendReportedWatermarkToAllSubtasks() {
+        for (int subtaskIndex = 0; subtaskIndex < numSubtasks; subtaskIndex++) {
+            sourceCoordinator.handleEventFromOperator(
+                    subtaskIndex, 0, new ReportedWatermarkEvent(generateTimestamp()));
+        }
+        waitForCoordinatorToProcessActions(sourceCoordinator.getContext());
+    }
+
+    private long generateTimestamp() {
+        switch (timestampType) {
+            case MONOTONE_INCREASING:
+                return System.currentTimeMillis();
+            case RANDOMIZE_MILLISECONDS:
+                return System.currentTimeMillis() / 1000 * 1000 + RANDOM.nextInt(1000);

Review Comment:
   Don't you actually benchmark here `RANDOM.nextInt()` and `System.currentTimeMillis()` instead of the `SourceCoordinator`? 🤔 The results you posted under:
   > The result of master branch with https://github.com/apache/flink/pull/22852:
   
   Suggest that this might be the case.
   
   `Random` is a very heavy class with lot's of overheads. `ThreadLocalRandom` is better, but there is no good replacement of `System.currentTimeMillis`, which is also quite heavy (syscall). 
   
   Usually the solution is have some pre-generated small array of timestamps in the `setup` method, that's later used in the benchmark. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] 1996fanrui commented on pull request #22921: [FLINK-32506][connectors/common] Add the benchmark for watermark aggregation when enabling the watermark alignment

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #22921:
URL: https://github.com/apache/flink/pull/22921#issuecomment-1614986008

   Hi @pnowojski  @RocMarshal , I have finished the first version of watermark aggregation benchmark. Would you mind help take a look in your free time? thanks~
   
   
   [Here](https://github.com/apache/flink-benchmarks/pull/77) is the PR of `flink-benchmark` repo.
   
   I run it on my local, and following is the benchmark result, the overall performance has been improved by more than 100 times(Some results are not stable, let's follow the official change from http://codespeed.dak8s.net:8000/ in the future).
   
   ```
   The result of master branch:
   Benchmark                                         (numSubtasks)         (timestampType)  Mode  Cnt     Score     Error  Units
   WatermarkAggregationBenchmark.aggregateWatermark           5000     MONOTONE_INCREASING  avgt   10   460.511 ±  31.441  ms/op
   WatermarkAggregationBenchmark.aggregateWatermark           5000  RANDOMIZE_MILLISECONDS  avgt   10   373.986 ±  25.024  ms/op
   WatermarkAggregationBenchmark.aggregateWatermark          10000     MONOTONE_INCREASING  avgt   10   856.788 ±  58.144  ms/op
   WatermarkAggregationBenchmark.aggregateWatermark          10000  RANDOMIZE_MILLISECONDS  avgt   10  1235.172 ± 114.965  ms/op
   WatermarkAggregationBenchmark.aggregateWatermark          20000     MONOTONE_INCREASING  avgt   10  5906.807 ±  91.858  ms/op
   WatermarkAggregationBenchmark.aggregateWatermark          20000  RANDOMIZE_MILLISECONDS  avgt   10  3318.771 ± 272.305  ms/op
   
   
   The result of master branch with https://github.com/apache/flink/pull/22852:
   Benchmark                                         (numSubtasks)         (timestampType)  Mode  Cnt   Score   Error  Units
   WatermarkAggregationBenchmark.aggregateWatermark           5000     MONOTONE_INCREASING  avgt   10   3.329 ± 0.070  ms/op
   WatermarkAggregationBenchmark.aggregateWatermark           5000  RANDOMIZE_MILLISECONDS  avgt   10   3.541 ± 0.048  ms/op
   WatermarkAggregationBenchmark.aggregateWatermark          10000     MONOTONE_INCREASING  avgt   10   7.345 ± 0.228  ms/op
   WatermarkAggregationBenchmark.aggregateWatermark          10000  RANDOMIZE_MILLISECONDS  avgt   10   7.620 ± 0.385  ms/op
   WatermarkAggregationBenchmark.aggregateWatermark          20000     MONOTONE_INCREASING  avgt   10  16.059 ± 0.307  ms/op
   WatermarkAggregationBenchmark.aggregateWatermark          20000  RANDOMIZE_MILLISECONDS  avgt   10  20.797 ± 0.818  ms/op
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org