You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "1996fanrui (via GitHub)" <gi...@apache.org> on 2023/10/19 09:51:38 UTC

[PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

1996fanrui opened a new pull request, #23550:
URL: https://github.com/apache/flink/pull/23550

   ## What is the purpose of the change
   
   See [FLINK-33315](https://issues.apache.org/jira/browse/FLINK-33315) for details.
   
   When creating a successor operator to a SourceOperator, the call stack is:
   
   - OperatorChain#createOperatorChain ->
   - wrapOperatorIntoOutput ->
   - getOperatorRecordsOutCounter ->
   - operatorConfig.getStreamOperatorFactory(userCodeClassloader)
   
   It will generate the SourceOperatorFactory temporarily and just check whether it's SinkWriterOperatorFactory
   
   This PR focus on avoid unnecessary heavy getStreamOperatorFactory, it can optimize the memory and cpu cost of Replica_2 in [FLINK-33315](https://issues.apache.org/jira/browse/FLINK-33315).
   
   This heavy operation is introduced in [FLINK-30536](https://issues.apache.org/jira/browse/FLINK-30536) (1.17), and it's not necessary. ([code link](https://github.com/apache/flink/blob/c2e14ff411e806f9ccf176c85eb8249b8ff12e56/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/OperatorChain.java#L646)) . 
   
   
   ## Brief change log
   
   Solution: We can store the serializedUdfClassName at StreamConfig, and using the getStreamOperatorFactoryClassName instead of the heavy getStreamOperatorFactory in OperatorChain#getOperatorRecordsOutCounter.
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency):  no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature?no
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "pnowojski (via GitHub)" <gi...@apache.org>.
pnowojski commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1371854774


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   Maybe solution 3 would be also fine. If it fails due to class loader issues, it would err on the safe side. 🤷 
   
   Solution 1 wouldn't be fragile in a sense, that it wouldn't allow for silent errors. As I mentioned, if you moved the class name equality check into `StreamConfig`'s method `boolean StreamConfig#isSinkWriterOperatorFactory(Class<...> ...)`, then both the equality check upon deserialization and the `checkState` during serialization would reside "close to one another" in the `StreamConfig` class. `checkState` would prevent from any silent issues. 
    



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   Maybe solution 3 would be also fine. If it fails due to class loader issues, it would err on the safe side. 🤷 
   
   Solution 1 wouldn't be fragile in a sense, that it wouldn't allow for silent errors. As I mentioned, if you moved the class name equality check into `StreamConfig`'s method `boolean StreamConfig#isSinkWriterOperatorFactory(Class<...> ...)`, then both the equality check upon deserialization and the `checkState` during serialization would reside "close to one another" in the `StreamConfig` class. `checkState` would prevent from any silent issues. 
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1371912000


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   Sorry @pnowojski , I misunderstood your solution1 before. I think I got it now.
   
   Let me express my understand:
   1. We still check the class name
   2. We added a check to prevent adding a sub class for SinkWriterOperatorFactory in the future.
   
   If it will be added in the future, this check will throw exception, and correponding developer will find it, right?
   
   If so, I think it's fine, and it's better to add some comments to explain why.
   
   Also, how about just adding the solution 1 to master branch? 1.17 and 1.18 don't merge new feature, and cuuerntly it's final class, so it's should be safe for 1.17 and 1.18.
   
   I will go ahead with solution1, thanks again for your suggestion and patient.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1371930593


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   I have a question, is there the same classloader issue for solution1?
   
   For example, the new sub class and SinkWriterOperatorFactory are  loaded by different classloader, and we cannot check whether the class is a subclass of SinkWriterOperatorFactory.
   
   Maybe my consideration is unnecessary, if the subclass cannot be check, too many code cannot work.
   
   Assume we can ensure the class loader is same, whether the solution3 is more clear? It allow developer add new subclass in the future, and code is easy to read.
   
   After I check the code, we can loader the SinkWriterOperatorFactory by userCodeClassLoader, so the class loader should be same.
   
   How about solution3?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 commented on PR #23550:
URL: https://github.com/apache/flink/pull/23550#issuecomment-1770896697

   @1996fanrui Thanks for the PR! LGTM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "pnowojski (via GitHub)" <gi...@apache.org>.
pnowojski commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1368508673


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   Maybe change the last line to:
   ```
   toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
   config.setString(IS_INSTANCE_OF_SinkWriterOperatorFactory, factory instance of SinkWriterOperatorFactory);
   ```
   ?



##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/OperatorChain.java:
##########
@@ -641,17 +641,15 @@ private Map<StreamConfig.SourceInputConfig, ChainedSource> createChainedSources(
     @Nullable
     private Counter getOperatorRecordsOutCounter(
             StreamTask<?, ?> containingTask, StreamConfig operatorConfig) {
-        ClassLoader userCodeClassloader = containingTask.getUserCodeClassLoader();
-        StreamOperatorFactory<?> operatorFactory =
-                operatorConfig.getStreamOperatorFactory(userCodeClassloader);
+        String streamOperatorFactoryClassName = operatorConfig.getStreamOperatorFactoryClassName();
         // Do not use the numRecordsOut counter on output if this operator is SinkWriterOperator.
         //
         // Metric "numRecordsOut" is defined as the total number of records written to the
         // external system in FLIP-33, but this metric is occupied in AbstractStreamOperator as the
         // number of records sent to downstream operators, which is number of Committable batches
         // sent to SinkCommitter. So we skip registering this metric on output and leave this metric
         // to sink writer implementations to report.
-        if (operatorFactory instanceof SinkWriterOperatorFactory) {
+        if (SinkWriterOperatorFactory.class.getName().equals(streamOperatorFactoryClassName)) {

Review Comment:
   @1996fanrui , are those actually the equivalent lines? What if the used `operatorFactory` is a subclass of `SinkWriterOperatorFactory`, with a different name? 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "lindong28 (via GitHub)" <gi...@apache.org>.
lindong28 merged PR #23550:
URL: https://github.com/apache/flink/pull/23550


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1371930593


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   I have a question, is there the same classloader issue for solution1?
   
   For example, the new sub class and SinkWriterOperatorFactory are  loaded by different classloader, and we cannot check whether the class is a subclass of SinkWriterOperatorFactory.
   
   Maybe my consideration is unnecessary, if the subclass cannot be check, too many code cannot work.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1371930593


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   I have a question, is there the same classloader issue for solution1?
   
   For example, the new sub class and SinkWriterOperatorFactory are  loaded by different classloader, and we cannot check whether the class is a subclass of SinkWriterOperatorFactory.
   
   Maybe my consideration is unnecessary, if the subclass cannot be check, too many code cannot work.
   
   Assume we can ignore the class loader issue. Whether the solution3 is more clear? It allow developer add new subclass in the future, and code is easy to read.
   
   Also, the TaskExextor and SinkWriterOperatorFactory should be loaded by one classloader, they are flink code, from the same dist jar. And we try to deserialize the operator factory using the TaskExector.class.getClassLoader, so the class loader should be same.
   
   How about solution3?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "pnowojski (via GitHub)" <gi...@apache.org>.
pnowojski commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1372805309


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   > If it will be added in the future, this check will throw exception, and correponding developer will find it, right?
   >
   > If so, I think it's fine, and it's better to add some comments to explain why.
   
   Yes :)
   
   > Also, how about just adding the solution 1 to master branch? 
   
   That's perfectly fine. In 1.17 1.18 branches it's extremely unlikely that anyone will ever introduce new sink factory.
   
   > I have a question, is there the same classloader issue for solution1?
   
   Have you noticed some issues? I wouldn't expect anything. As I said, a lot of things would brake down, if user class loader would have loaded a different `SinkWriterOperatorFactory` vs Flink's root class loader. It's not only that `instanceof` would fail, but you wouldn't be able to assign the user loaded class to many runtime variables/fields.
   
   First and foremost, user code shouldn't include SPI classes. Those dependencies should be declared as `provided`. Secondly, if I remember correctly, as a precaution we are using `ParentFirstClassLoader` for any Flink's classes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1371930593


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   I have a question, is there the same classloader issue for solution1?
   
   For example, the new sub class and SinkWriterOperatorFactory are  loaded by different classloader, and we cannot check whether the class is a subclass of SinkWriterOperatorFactory.
   
   Maybe my consideration is unnecessary, if the subclass cannot be check, too many code cannot work.
   
   Assume we can ensure the class loader is same, whether the solution3 is more clear? It allow developer add new subclass in the future, and code is easy to read.
   
   After I check the code[1],  we can load the SinkWriterOperatorFactory by userCodeClassLoader, and deserialize the operator factory by userCodeClassLoader, so the class loader should be same.
   
   How about solution3?
   
   [1] https://github.com/1996fanrui/flink/blob/1c7e5b6fcc9b16c66fb9722ab2da25dc87d0c4a4/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/OperatorChain.java#L652



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1371104358


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   Hi @pnowojski , thanks for your analysis!
   
   > I would move: SinkWriterOperatorFactory.class.getName().equals(streamOperatorFactoryClassName)
   > check, into the boolean StreamConfig#isSinkWriterOperatorFactory(Class<...> ...) method. 
   > It doesn't fit there very well, BUT at least it would justify why we have the checkState in the 
   > StreamConfig#setStreamOperatorFactory.
   
   The solution1 (`SinkWriterOperatorFactory.class.getName().equals(streamOperatorFactoryClassName)` ) is still fragile, right?
   
   When `makes SinkWriterOperatorFactory non final and implement a subclass` in the future, it still cannot support, and silently. 
   
   > toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
   config.setString(IS_INSTANCE_OF_SinkWriterOperatorFactory, factory instance of SinkWriterOperatorFactory);
   > would be better/cleaner. Either one is fine for me.
   
   The solution2 is perfectly compatible with the case of adding sub class.
   
   However, as I said before, `the getStreamOperatorFactory is called in the toString to print the class name.`, and I'd like to using the `SERIALIZED_UDF_CLASS_NAME` instead of `getStreamOperatorFactory`.
   
   If we just keep `IS_INSTANCE_OF_SinkWriterOperatorFactory`, we must call `getStreamOperatorFactory` in the `toString` method.
   
   Or we add the `IS_INSTANCE_OF_SinkWriterOperatorFactory` and `SERIALIZED_UDF_CLASS_NAME` together?
   
   
   Actually, I have solution3 before I create this PR:
   
   We store the `SERIALIZED_UDF_CLASS` instead of `SERIALIZED_UDF_CLASS_NAME`.
   
   ```
   # setStreamOperatorFactory method
   toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
   toBeSerializedConfigObjects.put(SERIALIZED_UDF_CLASS, factory.getClass());
   ```
   
   
   ```
       public <T extends StreamOperatorFactory<?>> Class<T> getStreamOperatorFactoryClass(ClassLoader cl) {
           try {
               return InstantiationUtil.readObjectFromConfig(this.config, SERIALIZED_UDF_CLASS, cl);
           } catch (Exception e) {
               throw new StreamTaskException("Could not instantiate chained outputs.", e);
           }
       }
   ```
   
   And check `isAssignableFrom`:
   
   ```
   SinkWriterOperatorFactory.class.isAssignableFrom(getStreamOperatorFactoryClass(SinkWriterOperatorFactory.class.getClassLoader()));
   ```
   
   The solution3 is fine, however, I'm worried that when there are multiple classloaders, the judgment may be wrong.
   
   That's why this PR store the ClassName instead of Class.
   
   
   WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1368530995


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/OperatorChain.java:
##########
@@ -641,17 +641,15 @@ private Map<StreamConfig.SourceInputConfig, ChainedSource> createChainedSources(
     @Nullable
     private Counter getOperatorRecordsOutCounter(
             StreamTask<?, ?> containingTask, StreamConfig operatorConfig) {
-        ClassLoader userCodeClassloader = containingTask.getUserCodeClassLoader();
-        StreamOperatorFactory<?> operatorFactory =
-                operatorConfig.getStreamOperatorFactory(userCodeClassloader);
+        String streamOperatorFactoryClassName = operatorConfig.getStreamOperatorFactoryClassName();
         // Do not use the numRecordsOut counter on output if this operator is SinkWriterOperator.
         //
         // Metric "numRecordsOut" is defined as the total number of records written to the
         // external system in FLIP-33, but this metric is occupied in AbstractStreamOperator as the
         // number of records sent to downstream operators, which is number of Committable batches
         // sent to SinkCommitter. So we skip registering this metric on output and leave this metric
         // to sink writer implementations to report.
-        if (operatorFactory instanceof SinkWriterOperatorFactory) {
+        if (SinkWriterOperatorFactory.class.getName().equals(streamOperatorFactoryClassName)) {

Review Comment:
   Thanks @pnowojski for the review!
   
   I see `SinkWriterOperatorFactory` is a final class for now. Are you worried about it being changed in the future?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #23550:
URL: https://github.com/apache/flink/pull/23550#issuecomment-1770475242

   Hi @lindong28 , this heavy operation is introduced in FLINK-30536, would you mind helping take a look this PR in your free time? thanks~ 
   
   cc @WeiZhong94 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #23550:
URL: https://github.com/apache/flink/pull/23550#issuecomment-1770961717

   Thanks to @lindong28 for the quick review and merging!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1368534908


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   Yes, it's more accurate. 
   
   Same reason to the last comment, `SinkWriterOperatorFactory` is final class. And the class name is more generic.
   
   Of course I can update if you think there might be a risk.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "pnowojski (via GitHub)" <gi...@apache.org>.
pnowojski commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1369745726


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   I've missed that it's a final class, thanks for pointing this out :) In that case, I think my only issue with this code is as you said, that it's fragile. If someone makes `SinkWriterOperatorFactory` non final and implement a subclass in one place of the system, suddenly completely different part of the code can fail, maybe even silently.
   
   However this could be protected via some `checkState()` in the `setStreamOperatorFactory`, that would asserting that factory is not a subclass of `SinkWriterOperatorFactory`.
   
   As this would be a bit hacky, I would move:
   ```
   SinkWriterOperatorFactory.class.getName().equals(streamOperatorFactoryClassName)
   ```
   check, into the `boolean StreamConfig#isSinkWriterOperatorFactory(Class<...> ...)` method. It doesn't fit there very well, BUT at least it would justify why we have the `checkState` in the `StreamConfig#setStreamOperatorFactory`.
   
   Or maybe:
   ```
   toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
   config.setString(IS_INSTANCE_OF_SinkWriterOperatorFactory, factory instance of SinkWriterOperatorFactory);
   ```
   would be better/cleaner. Either one is fine for me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1371930593


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   I have a question, is there the same classloader issue for solution1?
   
   For example, the new sub class and SinkWriterOperatorFactory are  loaded by different classloader, and we cannot check whether the class is a subclass of SinkWriterOperatorFactory.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1368541296


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   BTW, the `getStreamOperatorFactory` is called in the `toString` to print the class name.
   
   I'm think whether it should be replaced with the `CLASS_NAME`, if so, the `CLASS_NAME` is a extra benefit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "pnowojski (via GitHub)" <gi...@apache.org>.
pnowojski commented on code in PR #23550:
URL: https://github.com/apache/flink/pull/23550#discussion_r1369745726


##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java:
##########
@@ -368,7 +373,8 @@ public void setStreamOperator(StreamOperator<?> operator) {
 
     public void setStreamOperatorFactory(StreamOperatorFactory<?> factory) {
         if (factory != null) {
-            toBeSerializedConfigObjects.put(SERIALIZEDUDF, factory);
+            toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
+            config.setString(SERIALIZED_UDF_CLASS_NAME, factory.getClass().getName());

Review Comment:
   I've missed that it's a final class. In that case, I think my only issue with this code is as you said, that it's fragile. If someone makes `SinkWriterOperatorFactory` non final and implement a subclass in one place of the system, suddenly completely different part of the code can fail, maybe even silently.
   
   However this could be protected via some `checkState()` in the `setStreamOperatorFactory`, that would asserting that factory is not a subclass of `SinkWriterOperatorFactory`.
   
   As this would be a bit hacky, I would move:
   ```
   SinkWriterOperatorFactory.class.getName().equals(streamOperatorFactoryClassName)
   ```
   check, into the `boolean StreamConfig#isSinkWriterOperatorFactory(Class<...> ...)` method. It doesn't fit there very well, BUT at least it would justify why we have the `checkState` in the `StreamConfig#setStreamOperatorFactory`.
   
   Or maybe:
   ```
   toBeSerializedConfigObjects.put(SERIALIZED_UDF, factory);
   config.setString(IS_INSTANCE_OF_SinkWriterOperatorFactory, factory instance of SinkWriterOperatorFactory);
   ```
   would be better/cleaner. Either one is fine for me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33316][runtime] Avoid unnecessary heavy getStreamOperatorFactory [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #23550:
URL: https://github.com/apache/flink/pull/23550#issuecomment-1770477849

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c7e5b6fcc9b16c66fb9722ab2da25dc87d0c4a4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1c7e5b6fcc9b16c66fb9722ab2da25dc87d0c4a4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c7e5b6fcc9b16c66fb9722ab2da25dc87d0c4a4 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org