You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "1996fanrui (via GitHub)" <gi...@apache.org> on 2023/06/30 03:12:35 UTC

[GitHub] [flink] 1996fanrui opened a new pull request, #22911: [FLINK-32495][connectors/common] Fix the bug that the shared thread factory causes the source alignment unit test to fail

1996fanrui opened a new pull request, #22911:
URL: https://github.com/apache/flink/pull/22911

   ## What is the purpose of the change
   
   SourceCoordinatorAlignmentTest.testWatermarkAlignmentWithTwoGroups fails.
   
   I analyzed this CI :  https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50668&view=logs&j=a57e0635-3fad-5b08-57c7-a4142d7d6fa9&t=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7&l=9089
   
   Root cause:
   
   - The CoordinatorExecutorThreadFactory cannot new multiple threads. And too many callers will check `coordinatorThreadFactory.isCurrentThreadCoordinatorThread()`, such as: SourceCoordinatorContext.attemptReady.
   - The CoordinatorExecutorThreadFactory is shared at [SourceCoordinatorTestBase](https://github.com/apache/flink/blob/21eba4ca4cb235a2189c94cdbf3abcec5cde1e6e/flink-runtime/src/test/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorTestBase.java#L68)
   - It will be used at multiple source coordinator, and the second source coordinator will overwrite the CoordinatorExecutorThreadFactory#t, so the check will fail for the first source.
   
   ## Brief change log
   
   Don't share the CoordinatorExecutorThreadFactory.
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): ( no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no)
     - The serializers: (no)
     - The runtime per-record code paths (performance sensitive): (no)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (no)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] flinkbot commented on pull request #22911: [FLINK-32495][connectors/common] Fix the bug that the shared thread factory causes the source alignment unit test to fail

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #22911:
URL: https://github.com/apache/flink/pull/22911#issuecomment-1614061014

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3c4c80935861a6c7fa77547c42e5e3edd78a703f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c4c80935861a6c7fa77547c42e5e3edd78a703f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3c4c80935861a6c7fa77547c42e5e3edd78a703f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] RocMarshal commented on a diff in pull request #22911: [FLINK-32495][connectors/common] Fix the bug that the shared thread factory causes the source alignment unit test to fail

Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #22911:
URL: https://github.com/apache/flink/pull/22911#discussion_r1247548797


##########
flink-runtime/src/test/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorProviderTest.java:
##########
@@ -29,26 +29,25 @@ Licensed to the Apache Software Foundation (ASF) under one
 import org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator;
 import org.apache.flink.runtime.source.event.ReaderRegistrationEvent;
 
-import org.junit.Before;
-import org.junit.Test;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
 
 import java.time.Duration;
 import java.util.concurrent.CompletableFuture;
 
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertNotEquals;
-import static org.junit.Assert.assertTrue;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
 
 /** Unit tests for {@link SourceCoordinatorProvider}. */
-public class SourceCoordinatorProviderTest {
+class SourceCoordinatorProviderTest {
 
     private static final OperatorID OPERATOR_ID = new OperatorID(1234L, 5678L);
     private static final int NUM_SPLITS = 10;
 
     private SourceCoordinatorProvider<MockSourceSplit> provider;
 
-    @Before
-    public void setup() {
+    @BeforeEach
+    void setup() {
         provider =
                 new SourceCoordinatorProvider<>(
                         "SourceCoordinatorProviderTest",

Review Comment:
   How about
   
   ```suggestion
                           this.getClass().getSimpleName(),
   ```
   ?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorProvider.java:
##########
@@ -130,6 +135,10 @@ public static class CoordinatorExecutorThreadFactory
 
         @Override
         public synchronized Thread newThread(Runnable r) {
+            checkState(
+                    t == null,
+                    "Please using the new CoordinatorExecutorThreadFactory,"
+                            + " this factory cannot new multiple threads.");

Review Comment:
   nice~.
   This design is in line with the principle that It should fail first once encounter error operation.



##########
flink-runtime/src/test/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorProviderTest.java:
##########
@@ -125,4 +122,19 @@ public void testCallAsyncExceptionFailsJob() throws Exception {
                 Duration.ofSeconds(10L),
                 "The job did not fail before timeout.");
     }
+
+    @Test
+    void testCoordinatorExecutorThreadFactoryNewMultipleThread() {
+        SourceCoordinatorProvider.CoordinatorExecutorThreadFactory
+                coordinatorExecutorThreadFactory =
+                        new SourceCoordinatorProvider.CoordinatorExecutorThreadFactory(
+                                "test_coordinator_thread",
+                                new MockOperatorCoordinatorContext(
+                                        new OperatorID(1234L, 5678L), 3));
+
+        coordinatorExecutorThreadFactory.newThread(() -> {});
+        // coordinatorExecutorThreadFactory cannot create multiple threads.
+        assertThatThrownBy(() -> coordinatorExecutorThreadFactory.newThread(() -> {}))
+                .isInstanceOf(IllegalStateException.class);
+    }

Review Comment:
   👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] 1996fanrui commented on a diff in pull request #22911: [FLINK-32495][connectors/common] Fix the bug that the shared thread factory causes the source alignment unit test to fail

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #22911:
URL: https://github.com/apache/flink/pull/22911#discussion_r1247672074


##########
flink-runtime/src/test/java/org/apache/flink/runtime/source/coordinator/SourceCoordinatorProviderTest.java:
##########
@@ -29,26 +29,25 @@ Licensed to the Apache Software Foundation (ASF) under one
 import org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator;
 import org.apache.flink.runtime.source.event.ReaderRegistrationEvent;
 
-import org.junit.Before;
-import org.junit.Test;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
 
 import java.time.Duration;
 import java.util.concurrent.CompletableFuture;
 
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertNotEquals;
-import static org.junit.Assert.assertTrue;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
 
 /** Unit tests for {@link SourceCoordinatorProvider}. */
-public class SourceCoordinatorProviderTest {
+class SourceCoordinatorProviderTest {
 
     private static final OperatorID OPERATOR_ID = new OperatorID(1234L, 5678L);
     private static final int NUM_SPLITS = 10;
 
     private SourceCoordinatorProvider<MockSourceSplit> provider;
 
-    @Before
-    public void setup() {
+    @BeforeEach
+    void setup() {
         provider =
                 new SourceCoordinatorProvider<>(
                         "SourceCoordinatorProviderTest",

Review Comment:
   it's not related to this fix, and this fix is urgent, so i will merge it first.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] 1996fanrui commented on pull request #22911: [FLINK-32495][connectors/common] Fix the bug that the shared thread factory causes the source alignment unit test to fail

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #22911:
URL: https://github.com/apache/flink/pull/22911#issuecomment-1614407773

   Hi @snuyanzin  @RocMarshal , thanks for the quick review, merging.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] 1996fanrui merged pull request #22911: [FLINK-32495][connectors/common] Fix the bug that the shared thread factory causes the source alignment unit test to fail

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui merged PR #22911:
URL: https://github.com/apache/flink/pull/22911


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org