You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "snuyanzin (via GitHub)" <gi...@apache.org> on 2025/06/16 06:48:59 UTC

[PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

snuyanzin opened a new pull request, #26683:
URL: https://github.com/apache/flink/pull/26683

   
   ## What is the purpose of the change
   
   The PR makes it to not write empty lists to compiled plans
   
   
   
   ## Verifying this change
   This change is already covered by existing tests
   also added a test to ExecNodeVersionUpgradeSerdeTest
   
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): ( no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no)
     - The serializers: (no)
     - The runtime per-record code paths (performance sensitive): (no)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (no)
     - If yes, how is the feature documented? (not applicable)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on code in PR #26683:
URL: https://github.com/apache/flink/pull/26683#discussion_r2151405453


##########
flink-python/pyflink/table/tests/jsonplan/testGetJsonPlan.out:
##########
@@ -17,10 +17,8 @@
             }, {
               "name" : "c",
               "dataType" : "VARCHAR(2147483647)"
-            } ],
-            "watermarkSpecs" : [ ]
+            } ]
           },

Review Comment:
   if we remove this, how to distinguish between null and empty string. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26683:
URL: https://github.com/apache/flink/pull/26683#discussion_r2152328598


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/ExecNodeBase.java:
##########
@@ -116,7 +116,7 @@ protected ExecNodeBase(
             String description) {
         this.context = checkNotNull(context).withId(id);
         this.persistedConfig = persistedConfig == null ? new Configuration() : persistedConfig;
-        this.inputProperties = checkNotNull(inputProperties);
+        this.inputProperties = inputProperties == null ? List.of() : inputProperties;

Review Comment:
   only while reading from json when it is not present there
   however I checked and probably this could be replaced with annotation (added)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26683:
URL: https://github.com/apache/flink/pull/26683#issuecomment-2975312859

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d9331b68b6f8914302b91201a7229878c4970670",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d9331b68b6f8914302b91201a7229878c4970670",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d9331b68b6f8914302b91201a7229878c4970670 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26683:
URL: https://github.com/apache/flink/pull/26683#discussion_r2149653785


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/CompiledPlanSerdeUtil.java:
##########
@@ -265,6 +277,47 @@ static <T> Optional<T> deserializeOptionalField(
         return Optional.empty();
     }
 
+    static <T> List<T> deserializeList(
+            ObjectNode objectNode,
+            String fieldName,
+            Class<? extends T> type,
+            ObjectCodec codec,
+            DeserializationContext ctx)
+            throws IOException {
+        return ctx.readValue(
+                traverse(objectNode.required(fieldName), codec),
+                ctx.getTypeFactory().constructCollectionType(List.class, type));
+    }
+
+    static <T> List<T> deserializeListOrEmpty(
+            ObjectNode objectNode,
+            String fieldName,
+            Class<? extends T> type,
+            ObjectCodec codec,
+            DeserializationContext ctx)
+            throws IOException {
+        if (objectNode.hasNonNull(fieldName)) {
+            return deserializeList(objectNode, fieldName, type, codec, ctx);
+        }
+        return List.of();
+    }
+
+    static <K, V> Map<K, V> deserializeMapOrEmpty(

Review Comment:
   do you have example of it for compiled plan?
   I made only for those collections which I noticed
   Once any new collection/array appears there might be added a dedicated method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26683:
URL: https://github.com/apache/flink/pull/26683#discussion_r2149643993


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/CompiledPlanSerdeUtil.java:
##########
@@ -265,6 +277,49 @@ static <T> Optional<T> deserializeOptionalField(
         return Optional.empty();
     }
 
+    static <T> List<T> deserializeList(
+            ObjectNode objectNode,
+            String fieldName,
+            Class<? extends T> type,
+            ObjectCodec codec,
+            DeserializationContext ctx)
+            throws IOException {
+        return ctx.readValue(
+                traverse(objectNode.required(fieldName), codec),
+                ctx.getTypeFactory().constructCollectionType(List.class, type));
+    }
+
+    static <T> List<T> deserializeListOrEmpty(

Review Comment:
   Thanks - that makes sense. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #26683:
URL: https://github.com/apache/flink/pull/26683#issuecomment-2979079363

   it shouldn't have impact (it will continue working with compiled plans having this empty lists however new compiled plans will be without them), the purpose it to not put some useless info into compiled plan to make it a bit shorter. For huge queries it might play a role


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26683:
URL: https://github.com/apache/flink/pull/26683#discussion_r2149650394


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/CompiledPlanSerdeUtil.java:
##########
@@ -265,6 +277,47 @@ static <T> Optional<T> deserializeOptionalField(
         return Optional.empty();
     }
 
+    static <T> List<T> deserializeList(
+            ObjectNode objectNode,
+            String fieldName,
+            Class<? extends T> type,
+            ObjectCodec codec,
+            DeserializationContext ctx)
+            throws IOException {
+        return ctx.readValue(
+                traverse(objectNode.required(fieldName), codec),
+                ctx.getTypeFactory().constructCollectionType(List.class, type));
+    }
+
+    static <T> List<T> deserializeListOrEmpty(
+            ObjectNode objectNode,
+            String fieldName,
+            Class<? extends T> type,
+            ObjectCodec codec,
+            DeserializationContext ctx)
+            throws IOException {
+        if (objectNode.hasNonNull(fieldName)) {
+            return deserializeList(objectNode, fieldName, type, codec, ctx);
+        }
+        return List.of();
+    }
+
+    static <K, V> Map<K, V> deserializeMapOrEmpty(

Review Comment:
   I was curious do we need to do something about empty / null arrays, or is this handled already?  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on PR #26683:
URL: https://github.com/apache/flink/pull/26683#issuecomment-2976121295

   @snuyanzin I just approved assuming the CI failure is not related to the fix.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26683:
URL: https://github.com/apache/flink/pull/26683#discussion_r2149332176


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/CompiledPlanSerdeUtil.java:
##########
@@ -265,6 +277,49 @@ static <T> Optional<T> deserializeOptionalField(
         return Optional.empty();
     }
 
+    static <T> List<T> deserializeList(
+            ObjectNode objectNode,
+            String fieldName,
+            Class<? extends T> type,
+            ObjectCodec codec,
+            DeserializationContext ctx)
+            throws IOException {
+        return ctx.readValue(
+                traverse(objectNode.required(fieldName), codec),
+                ctx.getTypeFactory().constructCollectionType(List.class, type));
+    }
+
+    static <T> List<T> deserializeListOrEmpty(

Review Comment:
   Can you elaborate your question a bit ?
   Here it is an example of `deserializeList` usage (will fail if not present in json)
   https://github.com/apache/flink/pull/26683/files#diff-171890929b0275d27a8be97b4fd44f99d6910ce6649d1267c196b1285ced9ff3R65
   example of `deserializeListOrEmpty` (will be considered as empty if not present in json)
   https://github.com/apache/flink/pull/26683/files#diff-2f804b8ba94aa78460f122f44d050cd10af9386c78b9cd08edac79f4ef4fb00fR73
   
   and example of `deserializeMapOrEmpty`
   https://github.com/apache/flink/pull/26683/files#diff-3ce02724af75d34b80c98d3c52c33b6f88a81c0536beb6dd45e9a05a57abc1eeR73



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26683:
URL: https://github.com/apache/flink/pull/26683#discussion_r2149313798


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/CompiledPlanSerdeUtil.java:
##########
@@ -265,6 +277,49 @@ static <T> Optional<T> deserializeOptionalField(
         return Optional.empty();
     }
 
+    static <T> List<T> deserializeList(
+            ObjectNode objectNode,
+            String fieldName,
+            Class<? extends T> type,
+            ObjectCodec codec,
+            DeserializationContext ctx)
+            throws IOException {
+        return ctx.readValue(
+                traverse(objectNode.required(fieldName), codec),
+                ctx.getTypeFactory().constructCollectionType(List.class, type));
+    }
+
+    static <T> List<T> deserializeListOrEmpty(

Review Comment:
   I am curiou when does the caller use `deserializeList(` rather than `deserializeMapOrEmpty(`.  I assume in the  `deserializeList(` case  ` if (objectNode.hasNonNull(fieldName)) {` is always true. Do we need both methods?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on PR #26683:
URL: https://github.com/apache/flink/pull/26683#issuecomment-2976743867

   Thank you for taking a look @davidradl 
   the latest CI looks green


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26683:
URL: https://github.com/apache/flink/pull/26683#discussion_r2149313798


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/CompiledPlanSerdeUtil.java:
##########
@@ -265,6 +277,49 @@ static <T> Optional<T> deserializeOptionalField(
         return Optional.empty();
     }
 
+    static <T> List<T> deserializeList(
+            ObjectNode objectNode,
+            String fieldName,
+            Class<? extends T> type,
+            ObjectCodec codec,
+            DeserializationContext ctx)
+            throws IOException {
+        return ctx.readValue(
+                traverse(objectNode.required(fieldName), codec),
+                ctx.getTypeFactory().constructCollectionType(List.class, type));
+    }
+
+    static <T> List<T> deserializeListOrEmpty(

Review Comment:
   I am curious when does the caller use `deserializeList(` rather than `deserializeMapOrEmpty(`.  I assume in the  `deserializeList(` case  ` if (objectNode.hasNonNull(fieldName)) {` is always true. Do we need both methods?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #26683:
URL: https://github.com/apache/flink/pull/26683


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26683:
URL: https://github.com/apache/flink/pull/26683#discussion_r2151410039


##########
flink-python/pyflink/table/tests/jsonplan/testGetJsonPlan.out:
##########
@@ -17,10 +17,8 @@
             }, {
               "name" : "c",
               "dataType" : "VARCHAR(2147483647)"
-            } ],
-            "watermarkSpecs" : [ ]
+            } ]
           },

Review Comment:
   we do not have case for `watermarkSpecs` equal `null`, thus no need to distinguish, that's the thing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #26683:
URL: https://github.com/apache/flink/pull/26683#discussion_r2149362265


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/CompiledPlanSerdeUtil.java:
##########
@@ -265,6 +277,49 @@ static <T> Optional<T> deserializeOptionalField(
         return Optional.empty();
     }
 
+    static <T> List<T> deserializeList(
+            ObjectNode objectNode,
+            String fieldName,
+            Class<? extends T> type,
+            ObjectCodec codec,
+            DeserializationContext ctx)
+            throws IOException {
+        return ctx.readValue(
+                traverse(objectNode.required(fieldName), codec),
+                ctx.getTypeFactory().constructCollectionType(List.class, type));
+    }
+
+    static <T> List<T> deserializeListOrEmpty(

Review Comment:
   anyway i added some reuse



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on code in PR #26683:
URL: https://github.com/apache/flink/pull/26683#discussion_r2149677889


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/CompiledPlanSerdeUtil.java:
##########
@@ -265,6 +277,47 @@ static <T> Optional<T> deserializeOptionalField(
         return Optional.empty();
     }
 
+    static <T> List<T> deserializeList(
+            ObjectNode objectNode,
+            String fieldName,
+            Class<? extends T> type,
+            ObjectCodec codec,
+            DeserializationContext ctx)
+            throws IOException {
+        return ctx.readValue(
+                traverse(objectNode.required(fieldName), codec),
+                ctx.getTypeFactory().constructCollectionType(List.class, type));
+    }
+
+    static <T> List<T> deserializeListOrEmpty(
+            ObjectNode objectNode,
+            String fieldName,
+            Class<? extends T> type,
+            ObjectCodec codec,
+            DeserializationContext ctx)
+            throws IOException {
+        if (objectNode.hasNonNull(fieldName)) {
+            return deserializeList(objectNode, fieldName, type, codec, ctx);
+        }
+        return List.of();
+    }
+
+    static <K, V> Map<K, V> deserializeMapOrEmpty(

Review Comment:
   Hi @snuyanzin, no I did not have an example. Looking at the code I see you need map because of the options. I should have looked more closely.   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "liuyongvs (via GitHub)" <gi...@apache.org>.
liuyongvs commented on PR #26683:
URL: https://github.com/apache/flink/pull/26683#issuecomment-2979071706

   But this doesn't have any impact when having empty lists


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37957][table] Do not write empty lists to compiled plan [flink]

Posted by "twalthr (via GitHub)" <gi...@apache.org>.
twalthr commented on code in PR #26683:
URL: https://github.com/apache/flink/pull/26683#discussion_r2152075536


##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/ExecNodeBase.java:
##########
@@ -116,7 +116,7 @@ protected ExecNodeBase(
             String description) {
         this.context = checkNotNull(context).withId(id);
         this.persistedConfig = persistedConfig == null ? new Configuration() : persistedConfig;
-        this.inputProperties = checkNotNull(inputProperties);
+        this.inputProperties = inputProperties == null ? List.of() : inputProperties;

Review Comment:
   Annotate the arg if null is possible



##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/ExecNodeBase.java:
##########
@@ -116,7 +116,7 @@ protected ExecNodeBase(
             String description) {
         this.context = checkNotNull(context).withId(id);
         this.persistedConfig = persistedConfig == null ? new Configuration() : persistedConfig;
-        this.inputProperties = checkNotNull(inputProperties);
+        this.inputProperties = inputProperties == null ? List.of() : inputProperties;

Review Comment:
   when is null possible? add a comment for this case



##########
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/CompiledPlanSerdeUtil.java:
##########
@@ -278,6 +331,19 @@ static <T> Optional<T> deserializeOptionalField(
         return Optional.empty();
     }
 
+    static <T> T deserializeFieldOrNull(

Review Comment:
   ```suggestion
       static <T> @Nullable T deserializeFieldOrNull(
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org