You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "snuyanzin (via GitHub)" <gi...@apache.org> on 2026/07/29 09:14:43 UTC

[PR] [FLINK-40233][table] Correctly share the parsed JSON across `JSON_VALUE`/`JSON_QUERY` calls when the owning call is skipped because of `NULL` or not taken branch [flink]

snuyanzin opened a new pull request, #28836:
URL: https://github.com/apache/flink/pull/28836

   ## What is the purpose of the change
   
   It might happen that `NULL` is cached while it should not for instance in case of 
   ```sql
   
   SELECT json_value(v, cast(null as string)), json_value(v, '$.a')
   FROM (values('{"a":1, "b":2}')) AS t(v);
   -- returns (NULL, NULL), expected (NULL, 1)
   ```
   
   ## Brief change log
   
   codegen
   
   ## Verifying this change
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no)
     - The serializers: (no)
     - The runtime per-record code paths (performance sensitive): (no)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (no)
     - If yes, how is the feature documented? (not applicable)
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, please
   change the checkbox below to `[X]` followed by the name of the tool, and uncomment the
   "Generated-by" line. See the ASF Generative Tooling Guidance for details:
   https://www.apache.org/legal/generative-tooling.html
   
   You are responsible for the quality and correctness of every change in this PR
   regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
   AGENTS.md for the full guidance.
   -->
   
   - [ ] Yes (please specify the tool below)
   
   <!--
   Generated-by: [Tool Name and Version]
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-40233][table] Correctly share the parsed JSON across `JSON_VALUE`/`JSON_QUERY` calls when the owning call is skipped because of `NULL` or not taken branch [flink]

Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28836:
URL: https://github.com/apache/flink/pull/28836#discussion_r3675097602


##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/codegen/JsonParseReuseTest.java:
##########
@@ -165,4 +242,38 @@ void testDifferentJsonInputs() {
                 .as("JSON_VALUE calls on different inputs should parse separately")
                 .isEqualTo(2);
     }
+
+    @Test
+    void testReuseInOverWindowQueryIsResetPerRow() {
+        // JSON scalars run in a Calc ahead of the OverAggregate; each row must get its own parse
+        final TableEnvironment bEnv = TableEnvironment.create(EnvironmentSettings.inBatchMode());
+        bEnv.createTemporaryView(
+                "over_src",
+                bEnv.fromValues(Row.of(1, JSON_ROW1), Row.of(2, JSON_ROW2)).as("id", "j"));
+        final String sql =
+                "SELECT id, "
+                        + "MAX(JSON_VALUE(j, '$.type')) OVER (ORDER BY id ROWS BETWEEN CURRENT ROW AND CURRENT ROW), "
+                        + "MAX(JSON_QUERY(j, '$.address')) OVER (ORDER BY id ROWS BETWEEN CURRENT ROW AND CURRENT ROW) "
+                        + "FROM over_src";
+        final List<Row> rows = new ArrayList<>();
+        bEnv.executeSql(sql).collect().forEachRemaining(rows::add);
+        assertThat(rows)
+                .containsExactlyInAnyOrder(
+                        Row.of(1, "account", "{\"city\":\"Munich\"}"),
+                        Row.of(2, "admin", "{\"city\":\"Berlin\"}"));
+    }
+
+    @Test
+    void testFilterAndProjectionShareParse() {

Review Comment:
   All tests use a plain column: could you add a test using a computed column like using TRIM(json_data) as the input to the json functions? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-40233][table] Correctly share the parsed JSON across `JSON_VALUE`/`JSON_QUERY` calls when the owning call is skipped because of `NULL` or not taken branch [flink]

Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #28836:
URL: https://github.com/apache/flink/pull/28836


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-40233][table] Correctly share the parsed JSON across `JSON_VALUE`/`JSON_QUERY` calls when the owning call is skipped because of `NULL` or not taken branch [flink]

Posted by "wenshao (via GitHub)" <gi...@apache.org>.
wenshao commented on code in PR #28836:
URL: https://github.com/apache/flink/pull/28836#discussion_r3696464294


##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/JsonParseReuse.scala:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.table.planner.codegen.calls
+
+import org.apache.flink.table.planner.codegen.{CodeGeneratorContext, CodeGenUtils, GeneratedExpression}
+import org.apache.flink.table.planner.codegen.CodeGenUtils.qualifyMethod
+import org.apache.flink.table.runtime.functions.SqlJsonUtils
+
+/**
+ * Shares the parsed JSON input between the JSON function calls of a single generated expression,
+ * see [[JsonValueCallGen]] and [[JsonQueryCallGen]].
+ *
+ * The input is parsed at most once per record, by whichever call runs first, and the result is held
+ * in a member variable that the following calls on the same input read instead of parsing again.
+ */
+object JsonParseReuse {
+
+  /**
+   * Returns the expression holding the parsed input. The caller generates its own call with the
+   * original operands and reads the parsed input from the returned expression.
+   *
+   * The parse is lazy: the result term is a call to a member method that parses on its first
+   * invocation for the record. `generateCallWithStmtIfArgsNotNull` places a call under a guard that
+   * requires *all* of its arguments to be non-null, so no single call can be made the owner of the
+   * parse - in
+   * {{{
+   * SELECT JSON_VALUE(v, CAST(NULL AS STRING)), JSON_QUERY(v, '$.a')
+   * }}}
+   * the NULL path makes the first call short-circuit while the second one still needs the parse.
+   * Conversely, when no call runs, no parse happens at all.
+   */
+  def parseSharedInput(
+      ctx: CodeGeneratorContext,
+      operands: Seq[GeneratedExpression]): GeneratedExpression = {
+    val input = operands.head
+    val inputTerm = s"${input.resultTerm}.toString()"
+
+    ctx.getReusableInputUnboxingExprs(inputTerm, Int.MinValue) match {
+      case Some(expr) => expr
+      case None =>
+        val varName = CodeGenUtils.newName(ctx, "jsonParsed")
+        val lastInputName = CodeGenUtils.newName(ctx, "jsonParsedInput")
+        val methodName = CodeGenUtils.newName(ctx, "parseJson")
+        val typeName = classOf[SqlJsonUtils.JsonValueContext].getName
+        val inputType = CodeGenUtils.BINARY_STRING
+        ctx.addReusableMember(s"$typeName $varName;")
+        ctx.addReusableMember(s"$inputType $lastInputName;")
+
+        // keyed on the immutable input so it re-parses on change; no per-record reset needed

Review Comment:
   **[Suggestion]** The comment claims the cache is safe because the input is "immutable," but `BinaryStringData` is technically mutable (it extends `LazyBinaryFormat` which has a public `setJavaObject` setter and package-private mutable fields). The actual safety invariant is narrower: `BinaryRowData.getString()` creates a **new** `BinaryStringData` object per call (via `fromAddress` → `new BinaryStringData(...)`), so reference identity changes per record — Concrete cost: a future maintainer optimizing row unboxing may read "immutable" and conclude the cache is safe regardless of reference identity, or may "fix" `!=` to `!in.equals()`, replacing the O(1) identity check with an O(n) byte comparison on every JSON call site.
   
   ```suggestion
         // Reference-equality cache: BinaryRowData.getString() creates a fresh
         // BinaryStringData per call, so a new record always brings a new reference.
         // Do NOT switch to .equals() — the O(1) identity check is the point.
   ```
   
   _— qwen3.8-max-preview via Qwen Code /review (v0.21.2)_



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-40233][table] Correctly share the parsed JSON across `JSON_VALUE`/`JSON_QUERY` calls when the owning call is skipped because of `NULL` or not taken branch [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28836:
URL: https://github.com/apache/flink/pull/28836#issuecomment-5115836356

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bd385c6d1553c189022326a812d4e028a10a5301",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bd385c6d1553c189022326a812d4e028a10a5301",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bd385c6d1553c189022326a812d4e028a10a5301 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-40233][table] Correctly share the parsed JSON across `JSON_VALUE`/`JSON_QUERY` calls when the owning call is skipped because of `NULL` or not taken branch [flink]

Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28836:
URL: https://github.com/apache/flink/pull/28836#discussion_r3675065120


##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/codegen/JsonParseReuseTest.java:
##########
@@ -149,9 +154,81 @@ void testThreeJsonFunctionCalls() {
                 .containsExactlyInAnyOrder(
                         Row.of("account", "42", "{\"city\":\"Munich\"}"),
                         Row.of("admin", "30", "{\"city\":\"Berlin\"}"));
-        assertThat(countJsonParse(extractGeneratedCode(sql)))
+        final String code = extractGeneratedCode(sql);
+        assertThat(countJsonParse(code))
                 .as("Three JSON function calls on the same input should parse once")
-                .isEqualTo(1);
+                .isOne();
+    }
+
+    @Test
+    void testReuseSurvivesCodeSplitting() {

Review Comment:
   Are you sure this is testing code splitting? 
   
   extractGeneratedCode reads GeneratedClass.getCode(), which is the original source; splitting is applied in the constructor into the separate splitCode field and only that one is compiled (GeneratedClass.java:55-62, :97). The config only affects the collect(sql) half. I think it needs some tweaks in the test to test splitting
   
   But to be honest is not the most important thing to test in the PR, so I'd not say this is a blocker and dropping the test would also be ok 



##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/codegen/JsonParseReuseTest.java:
##########
@@ -133,9 +137,10 @@ void testJsonValueAndJsonQueryMixed() {
                 .containsExactlyInAnyOrder(
                         Row.of("account", "{\"city\":\"Munich\"}"),
                         Row.of("admin", "{\"city\":\"Berlin\"}"));
-        assertThat(countJsonParse(extractGeneratedCode(sql)))
+        final String code = extractGeneratedCode(sql);
+        assertThat(countJsonParse(code))
                 .as("JSON_VALUE + JSON_QUERY on the same input should parse once")
-                .isEqualTo(1);
+                .isOdd();

Review Comment:
   Leftover?
   
   ```suggestion
                   .isOne();
   ```



##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/codegen/JsonParseReuseTest.java:
##########
@@ -165,4 +242,38 @@ void testDifferentJsonInputs() {
                 .as("JSON_VALUE calls on different inputs should parse separately")
                 .isEqualTo(2);
     }
+

Review Comment:
   I think I found one edge case, but it might have an easy solution
   
   Repro
   ```
   -- rows: (1000,'{"n":1}') (2000,'{"n":2}') (3000,'{"n":3}') (9000,'{"n":9}')
   SELECT total FROM events MATCH_RECOGNIZE (
     ORDER BY rt
     MEASURES SUM(CAST(JSON_VALUE(A.f1, '$.n') AS INT)) AS total
     AFTER MATCH SKIP PAST LAST ROW
     PATTERN (A+ B)
     DEFINE A AS A.f0 < 9000, B AS B.f0 >= 9000
   )
   -- this branch: [3]  (= 1+1+1)
   -- master:      [6]  (= 1+2+3)
   ```
   
   The aggregate loops over every row of the match (MatchCodeGenerator.scala), but jsonParsed$0 = null; is a per-record statement, emitted once per processMatch - so rows 2..n reuse row 1's parse. Same cause in batch with table.exec.operator-fusion-codegen.enabled=true: CalcFusionCodegenSpec never emits reusePerRecordCode(), so the reset is dropped entirely.
   
   Master is safe because the assignment was unconditional at the call site. The problem is that correctness now depends on which operator template wraps the expression. 
   
   Possible easy solution: A local (addReusableLocalVariable, plus an explicit = null init) would fix both - reuseLocalVariableCode() is emitted in the fusion path and is keyed per method, so it resets inside the MATCH loop too



##########
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/JsonParseReuse.scala:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.table.planner.codegen.calls
+
+import org.apache.flink.table.planner.codegen.{CodeGeneratorContext, CodeGenUtils, GeneratedExpression}
+import org.apache.flink.table.planner.codegen.CodeGenUtils.qualifyMethod
+import org.apache.flink.table.runtime.functions.SqlJsonUtils
+
+/**
+ * Shares the parsed JSON input between the JSON function calls of a single generated expression,
+ * see [[JsonValueCallGen]] and [[JsonQueryCallGen]].
+ *
+ * The input is parsed at most once per record, by whichever call runs first, and the result is held
+ * in a member variable that the following calls on the same input read instead of parsing again.
+ */
+object JsonParseReuse {
+
+  /**
+   * Returns the expression holding the parsed input. The caller generates its own call with the
+   * original operands and reads the parsed input from the returned expression.
+   *
+   * The parse is lazy: the result term is a call to a member method that parses on its first
+   * invocation for the record. `generateCallWithStmtIfArgsNotNull` places a call under a guard that
+   * requires *all* of its arguments to be non-null, so no single call can be made the owner of the
+   * parse - in
+   * {{{
+   * SELECT JSON_VALUE(v, CAST(NULL AS STRING)), JSON_QUERY(v, '$.a')
+   * }}}
+   * the NULL path makes the first call short-circuit while the second one still needs the parse.
+   * Conversely, when no call runs, no parse happens at all.
+   */
+  def parseSharedInput(
+      ctx: CodeGeneratorContext,
+      operands: Seq[GeneratedExpression]): GeneratedExpression = {
+    val input = operands.head
+    val inputTerm = s"${input.resultTerm}.toString()"
+
+    ctx.getReusableInputUnboxingExprs(inputTerm, Int.MinValue) match {
+      case Some(expr) => expr
+      case None =>
+        val varName = CodeGenUtils.newName(ctx, "jsonParsed")
+        val methodName = CodeGenUtils.newName(ctx, "parseJson")
+        val typeName = classOf[SqlJsonUtils.JsonValueContext].getName
+        ctx.addReusableMember(s"$typeName $varName;")
+
+        // null means not parsed yet - jsonParse never returns null for a non-null input
+        ctx.addReusableMember(
+          s"""
+             |private $typeName $methodName(Object in) {

Review Comment:
   Could we do this? 
   
   ```suggestion
                |private $typeName $methodName(${CodeGenUtils.BINARY_STRING} in) {
   ```



##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/codegen/JsonParseReuseTest.java:
##########
@@ -165,4 +242,38 @@ void testDifferentJsonInputs() {
                 .as("JSON_VALUE calls on different inputs should parse separately")
                 .isEqualTo(2);
     }
+
+    @Test
+    void testReuseInOverWindowQueryIsResetPerRow() {
+        // JSON scalars run in a Calc ahead of the OverAggregate; each row must get its own parse
+        final TableEnvironment bEnv = TableEnvironment.create(EnvironmentSettings.inBatchMode());
+        bEnv.createTemporaryView(
+                "over_src",
+                bEnv.fromValues(Row.of(1, JSON_ROW1), Row.of(2, JSON_ROW2)).as("id", "j"));
+        final String sql =
+                "SELECT id, "
+                        + "MAX(JSON_VALUE(j, '$.type')) OVER (ORDER BY id ROWS BETWEEN CURRENT ROW AND CURRENT ROW), "
+                        + "MAX(JSON_QUERY(j, '$.address')) OVER (ORDER BY id ROWS BETWEEN CURRENT ROW AND CURRENT ROW) "
+                        + "FROM over_src";
+        final List<Row> rows = new ArrayList<>();
+        bEnv.executeSql(sql).collect().forEachRemaining(rows::add);
+        assertThat(rows)
+                .containsExactlyInAnyOrder(
+                        Row.of(1, "account", "{\"city\":\"Munich\"}"),
+                        Row.of(2, "admin", "{\"city\":\"Berlin\"}"));
+    }
+
+    @Test
+    void testFilterAndProjectionShareParse() {

Review Comment:
   All tests use a plain column: could you add a test using a computed column like using UPPER(json_data) as the input to the json functions?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-40233][table] Correctly share the parsed JSON across `JSON_VALUE`/`JSON_QUERY` calls when the owning call is skipped because of `NULL` or not taken branch [flink]

Posted by "gustavodemorais (via GitHub)" <gi...@apache.org>.
gustavodemorais commented on code in PR #28836:
URL: https://github.com/apache/flink/pull/28836#discussion_r3675065120


##########
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/codegen/JsonParseReuseTest.java:
##########
@@ -149,9 +154,81 @@ void testThreeJsonFunctionCalls() {
                 .containsExactlyInAnyOrder(
                         Row.of("account", "42", "{\"city\":\"Munich\"}"),
                         Row.of("admin", "30", "{\"city\":\"Berlin\"}"));
-        assertThat(countJsonParse(extractGeneratedCode(sql)))
+        final String code = extractGeneratedCode(sql);
+        assertThat(countJsonParse(code))
                 .as("Three JSON function calls on the same input should parse once")
-                .isEqualTo(1);
+                .isOne();
+    }
+
+    @Test
+    void testReuseSurvivesCodeSplitting() {

Review Comment:
   Are you sure this is testing code splitting? 
   
   extractGeneratedCode reads GeneratedClass.getCode(), which is the original source; splitting is applied in the constructor into the separate splitCode field and only that one is compiled (GeneratedClass.java:55-62, :97). The config only affects the collect(sql) half. I think it needs some tweaks in the test to test splitting
   
   But to be honest is not the most important thing to test in the PR, so I'd not say this is a blocker for me and dropping the test would also be ok 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org