You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "haruki-830 (via GitHub)" <gi...@apache.org> on 2026/07/15 09:57:57 UTC
[PR] [FLINK-40151][runtime] Align YAML Transform BETWEEN, IN, and LIKE semantics with Flink SQL [flink-cdc]
haruki-830 opened a new pull request, #4474:
URL: https://github.com/apache/flink-cdc/pull/4474
### Summary
This commit aligns YAML Transform row-level predicate semantics with Flink SQL. Predicates now preserve SQL `UNKNOWN` as Java `Boolean null`, filters only pass `TRUE`, and SQL pattern matching no longer relies on Java regex behavior.
### Key Changes
**SQL Three-Valued Logic**
- Updated comparison predicates to return `UNKNOWN` when operands contain `NULL`.
- Covers `=`, `<>`, `<`, `<=`, `>`, `>=`.
- Added SQL-compatible `AND`, `OR`, and `NOT` handling for `Boolean null`.
**BETWEEN and IN Semantics**
- Updated `BETWEEN` / `NOT BETWEEN` to preserve `UNKNOWN` according to SQL logic.
- Updated `IN` / `NOT IN` so `NULL` operands or candidates produce SQL-compatible results.
**LIKE and SIMILAR TO Semantics**
- Updated `LIKE` / `NOT LIKE` to use SQL wildcard semantics.
- Supports `%`, `_`, and `LIKE ... ESCAPE`.
- Added support for `SIMILAR TO` / `NOT SIMILAR TO`.
- Returns `UNKNOWN` when relevant arguments are `NULL`.
**Additional Predicate Support**
- Added support for `IS UNKNOWN` / `IS NOT UNKNOWN`.
- Added support for `IS DISTINCT FROM` / `IS NOT DISTINCT FROM`.
- Kept boolean condition consumers null-safe so `UNKNOWN` does not cause Java unboxing failures.
**Test Coverage**
- Added unit tests for comparison and string predicate functions.
- Updated parser tests for generated Janino expressions.
- Updated YAML transform specs for NULL, LIKE, ESCAPE, SIMILAR TO, and distinct/unknown predicates.
**JIRA Reference**
[https://issues.apache.org/jira/browse/FLINK-40151](https://issues.apache.org/jira/browse/FLINK-40151)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#discussion_r3621066520
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##########
@@ -322,6 +334,27 @@ private static Java.Rvalue generateUnaryOperation(
return new Java.UnaryOperation(Location.NOWHERE, operator, atom);
}
+ private static Java.Rvalue generateFunctionOperation(String functionName, Java.Rvalue[] atoms) {
+ return new Java.MethodInvocation(Location.NOWHERE, null, functionName, atoms);
+ }
+
+ private static Java.Rvalue generateLazyBinaryFunctionOperation(
+ Context context, SqlBasicCall sqlBasicCall, String functionName, Java.Rvalue[] atoms) {
+ if (atoms.length != 2) {
+ throw new ParseException("Unrecognized expression: " + sqlBasicCall.toString());
+ }
+ Java.Rvalue rightOperandSupplier =
+ new Java.AmbiguousName(
+ Location.NOWHERE,
+ new String[] {
+ "new java.util.function.Supplier<Boolean>() { public Boolean get() { return "
Review Comment:
We may add a test spec like `true OR 1/0`, `false AND 1/0` to ensure the short-circuit logic is effective.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation [flink-cdc]
Posted by "haruki-830 (via GitHub)" <gi...@apache.org>.
haruki-830 commented on code in PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#discussion_r3621716275
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##########
@@ -322,6 +334,27 @@ private static Java.Rvalue generateUnaryOperation(
return new Java.UnaryOperation(Location.NOWHERE, operator, atom);
}
+ private static Java.Rvalue generateFunctionOperation(String functionName, Java.Rvalue[] atoms) {
+ return new Java.MethodInvocation(Location.NOWHERE, null, functionName, atoms);
+ }
+
+ private static Java.Rvalue generateLazyBinaryFunctionOperation(
+ Context context, SqlBasicCall sqlBasicCall, String functionName, Java.Rvalue[] atoms) {
+ if (atoms.length != 2) {
+ throw new ParseException("Unrecognized expression: " + sqlBasicCall.toString());
+ }
+ Java.Rvalue rightOperandSupplier =
+ new Java.AmbiguousName(
+ Location.NOWHERE,
+ new String[] {
+ "new java.util.function.Supplier<Boolean>() { public Boolean get() { return "
Review Comment:
Added them in the latest commit. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Align YAML Transform BETWEEN, IN, and LIKE semantics with Flink SQL [flink-cdc]
Posted by "haruki-830 (via GitHub)" <gi...@apache.org>.
haruki-830 commented on PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#issuecomment-4998296354
> Also, to ease review, consider splitting this so buckets 1–2 and bucket 3 land as separate commits (or separate PRs) — the crash/parse fixes can then be merged quickly, while the semantic change is reviewed on its own.
Thanks for the detailed review and the compatibility analysis. I agree with the split.
I will separate this work into two PRs:
1. The first PR will only include the always-on changes with no silent behavior change:
- newly supported operators that are currently unparseable, such as `IS [NOT] DISTINCT FROM`, `IS [NOT] UNKNOWN`, `SIMILAR TO` / `NOT SIMILAR TO`, and `LIKE ... ESCAPE`
- NULL boolean crash fixes for `AND`, `OR`, `NOT`, `IS [NOT] TRUE`, and `IS [NOT] FALSE`
2. The second PR will handle the backward-incompatible semantic changes behind an explicit option:
- comparison predicates, `IN`, `BETWEEN`, and legacy `LIKE` / `NOT LIKE`
- add `pipeline.transform.expression.semantics` with `LEGACY` as the default and `FLINK_SQL` as the opt-in SQL-compatible behavior
This should allow the parse/crash fixes to move forward independently, while keeping the silent result-changing semantics isolated and explicitly gated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Align YAML Transform BETWEEN, IN, and LIKE semantics with Flink SQL [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#issuecomment-4993513284
Thanks for the work here. The changes split into three buckets with very different compatibility impact. I'd suggest applying buckets 1–2 unconditionally and gating bucket 3 behind an explicit option.
**1. New operators (currently unparseable) — keep always-on**
SQL: `IS [NOT] DISTINCT FROM`, `IS [NOT] UNKNOWN`, `SIMILAR TO` / `NOT SIMILAR TO`, `LIKE ... ESCAPE '<char>'`.
These fail validation on the current release, so no existing pipeline can depend on them. Safe to enable unconditionally.
**2. Crash → 3-valued logic, non-NULL behavior unchanged — keep always-on**
SQL: `AND`, `OR`, `NOT`, `IS [NOT] TRUE`, `IS [NOT] FALSE` where the operand is a NULL boolean (nullable `BOOLEAN` column or `CAST(NULL AS BOOLEAN)`).
These previously threw an NPE on NULL; results for non-NULL operands are identical. Pure crash fix, safe to enable unconditionally.
**3. Silent result changes — must be gated behind an explicit option**
SQL:
- Comparison / set predicates: `=`, `<>`, `>`, `>=`, `<`, `<=`, `IN`, `NOT IN`, `BETWEEN`, `NOT BETWEEN`. On a NULL operand the result changes from `false`/`true` to `NULL`. For negated forms (`<>`, `NOT IN`, `NOT BETWEEN`) this is `true → NULL`, which **drops rows in `WHERE`** and **flips `CASE` / `IF` branches**.
- `LIKE` / `NOT LIKE`: pattern engine changes from Java regex (`Pattern.find()`, substring match) to SQL wildcards (`%`, `_`, whole-string match). This changes results for **every row, not just NULL** — e.g. `'xabcy' LIKE 'abc'` goes `true → false`, `col LIKE 'A%'` goes `false → true`.
This is a backward-incompatible change to user-facing semantics, and the silent nature makes it dangerous: after the upgrade the YAML is unchanged, the job stays healthy, and no error or warning is emitted, yet the output data changes. For a CDC sync, altered filter results change which `INSERT` / `UPDATE` / `DELETE` events reach the sink, so the sink silently diverges from the source with no signal to detect or repair it. It therefore must not be the default on a minor/patch release.
**Recommendation**
Introduce `pipeline.transform.expression.semantics` (enum `LEGACY` | `FLINK_SQL`, default `LEGACY`). Keep the current behavior under `LEGACY`; apply bucket 3 only under `FLINK_SQL`. Buckets 1 and 2 remain always-on.
Also, to ease review, consider splitting this so buckets 1–2 and bucket 3 land as separate commits (or separate PRs) — the crash/parse fixes can then be merged quickly, while the semantic change is reviewed on its own.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation [flink-cdc]
Posted by "haruki-830 (via GitHub)" <gi...@apache.org>.
haruki-830 commented on code in PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#discussion_r3610033938
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##########
@@ -322,6 +334,21 @@ private static Java.Rvalue generateUnaryOperation(
return new Java.UnaryOperation(Location.NOWHERE, operator, atom);
}
+ private static Java.Rvalue generateFunctionOperation(String functionName, Java.Rvalue[] atoms) {
+ return new Java.MethodInvocation(Location.NOWHERE, null, functionName, atoms);
+ }
+
+ private static Java.Rvalue generateLazyBinaryFunctionOperation(
+ Context context, SqlBasicCall sqlBasicCall, String functionName, Java.Rvalue[] atoms) {
+ if (atoms.length != 2) {
+ throw new ParseException("Unrecognized expression: " + sqlBasicCall.toString());
+ }
+ Java.Rvalue rightOperandSupplier =
+ new Java.AmbiguousName(Location.NOWHERE, new String[] {"() -> " + atoms[1]});
Review Comment:
Addressed in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on code in PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#discussion_r3602147485
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##########
@@ -322,6 +334,21 @@ private static Java.Rvalue generateUnaryOperation(
return new Java.UnaryOperation(Location.NOWHERE, operator, atom);
}
+ private static Java.Rvalue generateFunctionOperation(String functionName, Java.Rvalue[] atoms) {
+ return new Java.MethodInvocation(Location.NOWHERE, null, functionName, atoms);
+ }
+
+ private static Java.Rvalue generateLazyBinaryFunctionOperation(
+ Context context, SqlBasicCall sqlBasicCall, String functionName, Java.Rvalue[] atoms) {
+ if (atoms.length != 2) {
+ throw new ParseException("Unrecognized expression: " + sqlBasicCall.toString());
+ }
+ Java.Rvalue rightOperandSupplier =
+ new Java.AmbiguousName(Location.NOWHERE, new String[] {"() -> " + atoms[1]});
Review Comment:
Could we add an end-to-end compilation test to `JaninoCompilerTest`?
The existing tests only verify the generated expression string or invoke
`LogicalFunctions` directly. They do not compile the generated expression with
Janino, so they cannot detect that Janino 3.1.10 does not support lambda
expressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#issuecomment-5002148059
Please update both the English and Chinese transform documentation to reflect these changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#issuecomment-5019334201
Also cc’ing @yuxiqian.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation [flink-cdc]
Posted by "haruki-830 (via GitHub)" <gi...@apache.org>.
haruki-830 commented on PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#issuecomment-5031890609
@lvyanquan Thanks for the detailed feedback.
I split the follow-up work into separate JIRA issues:
- https://issues.apache.org/jira/browse/FLINK-40203
- https://issues.apache.org/jira/browse/FLINK-40204
For this PR, I kept the scope limited to the always-on changes that do not introduce silent behavior changes. The backward-incompatible semantic alignment work, such as comparison / IN / BETWEEN / two-argument LIKE behavior, will be handled under the explicit semantics option in a follow-up PR.
For nullable AND / OR, I applied the localized optimization suggested here: native `&&` / `||` for non-null operands, conditional expressions when only the left operand is non-null, and the existing lazy fallback when the left operand is nullable. Avoiding that fallback completely would require statement-level codegen, so I tracked it separately instead of expanding this PR into an expression translator refactor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on code in PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#discussion_r3619220940
##########
flink-cdc-composer/src/test/resources/specs/comparison.yaml:
##########
@@ -165,6 +165,26 @@
DataChangeEvent{tableId=foo.bar.baz, before=[-1, 天地玄黄宇宙洪荒, true, true, true, false], after=[], op=DELETE, meta=()}
DataChangeEvent{tableId=foo.bar.baz, before=[], after=[0, null, true, false, true, false], op=INSERT, meta=()}
DataChangeEvent{tableId=foo.bar.baz, before=[0, null, true, false, true, false], after=[], op=DELETE, meta=()}
+- do: IsUnknown Op
+ projection: id_, bool_, bool_ IS UNKNOWN AS comp_1, bool_ IS NOT UNKNOWN AS comp_2, (bool_ AND CAST(NULL AS BOOLEAN)) IS UNKNOWN AS comp_3
+ primary-key: id_
+ expect: |-
+ CreateTableEvent{tableId=foo.bar.baz, schema=columns={`id_` BIGINT NOT NULL 'Identifier',`bool_` BOOLEAN 'George' 'false',`comp_1` BOOLEAN,`comp_2` BOOLEAN,`comp_3` BOOLEAN}, primaryKeys=id_, options=()}
+ DataChangeEvent{tableId=foo.bar.baz, before=[], after=[1, true, false, true, true], op=INSERT, meta=()}
+ DataChangeEvent{tableId=foo.bar.baz, before=[1, true, false, true, true], after=[-1, false, false, true, false], op=UPDATE, meta=()}
+ DataChangeEvent{tableId=foo.bar.baz, before=[-1, false, false, true, false], after=[], op=DELETE, meta=()}
+ DataChangeEvent{tableId=foo.bar.baz, before=[], after=[0, null, true, false, true], op=INSERT, meta=()}
+ DataChangeEvent{tableId=foo.bar.baz, before=[0, null, true, false, true], after=[], op=DELETE, meta=()}
+- do: Distinct From Op
+ projection: id_, int_ IS DISTINCT FROM 4 AS comp_1, int_ IS NOT DISTINCT FROM 4 AS comp_2, CAST(NULL AS INT) IS DISTINCT FROM int_ AS comp_3, CAST(NULL AS INT) IS NOT DISTINCT FROM int_ AS comp_4
+ primary-key: id_
+ expect: |-
+ CreateTableEvent{tableId=foo.bar.baz, schema=columns={`id_` BIGINT NOT NULL 'Identifier',`comp_1` BOOLEAN,`comp_2` BOOLEAN,`comp_3` BOOLEAN,`comp_4` BOOLEAN}, primaryKeys=id_, options=()}
+ DataChangeEvent{tableId=foo.bar.baz, before=[], after=[1, false, true, true, false], op=INSERT, meta=()}
+ DataChangeEvent{tableId=foo.bar.baz, before=[1, false, true, true, false], after=[-1, true, false, true, false], op=UPDATE, meta=()}
+ DataChangeEvent{tableId=foo.bar.baz, before=[-1, true, false, true, false], after=[], op=DELETE, meta=()}
+ DataChangeEvent{tableId=foo.bar.baz, before=[], after=[0, true, false, false, true], op=INSERT, meta=()}
+ DataChangeEvent{tableId=foo.bar.baz, before=[0, true, false, false, true], after=[], op=DELETE, meta=()}
Review Comment:
Lacks end-to-end tests for:
- NOT LIKE ... ESCAPE
- SIMILAR TO ... ESCAPE
- NOT SIMILAR TO ... ESCAPE
- NULL inputs for these expressions
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##########
@@ -322,6 +334,27 @@ private static Java.Rvalue generateUnaryOperation(
return new Java.UnaryOperation(Location.NOWHERE, operator, atom);
}
+ private static Java.Rvalue generateFunctionOperation(String functionName, Java.Rvalue[] atoms) {
+ return new Java.MethodInvocation(Location.NOWHERE, null, functionName, atoms);
+ }
+
+ private static Java.Rvalue generateLazyBinaryFunctionOperation(
+ Context context, SqlBasicCall sqlBasicCall, String functionName, Java.Rvalue[] atoms) {
+ if (atoms.length != 2) {
+ throw new ParseException("Unrecognized expression: " + sqlBasicCall.toString());
+ }
+ Java.Rvalue rightOperandSupplier =
+ new Java.AmbiguousName(
+ Location.NOWHERE,
+ new String[] {
+ "new java.util.function.Supplier<Boolean>() { public Boolean get() { return "
Review Comment:
A possible optimization is to generate different code based on operand nullability:
- If both operands are non-null, use native &&/||.
- If only the left operand is non-null, use a conditional expression:
- AND: left ? right : Boolean.FALSE
- OR: left ? Boolean.TRUE : right
- If the left operand is nullable, keep the current Supplier fallback to preserve three-valued logic and short-circuit evaluation.
This avoids unnecessary per-record Supplier allocations in common non-null cases.
##########
docs/content.zh/docs/core-concept/transform.md:
##########
@@ -120,8 +120,8 @@ Flink CDC 使用 [Calcite](https://calcite.apache.org/) 来解析表达式并且
| value1 >= value2 | greaterThanOrEqual(value1, value2) | 如果 value1 大于等于 value2，返回 TRUE；如果 value1 或 value2 为 NULL，返回 FALSE。 |
| value1 < value2 | lessThan(value1, value2) | 如果 value1 小于 value2，返回 TRUE；如果 value1 或 value2 为 NULL，返回 FALSE。 |
| value1 <= value2 | lessThanOrEqual(value1, value2) | 如果 value1 小于等于 value2，返回 TRUE；如果 value1 或 value2 为 NULL，返回 FALSE。 |
-| value IS NULL | null == value | 如果 value 为 NULL，返回 TRUE。 |
-| value IS NOT NULL | null != value | 如果 value 不为 NULL，返回 TRUE。 |
+| value IS NULL | isNull(value) | 如果 value 为 NULL，返回 TRUE。 |
+| value IS NOT NULL | isNotNull(value) | 如果 value 不为 NULL，返回 TRUE。 |
Review Comment:
The following functions are not documented:
- IS DISTINCT FROM
- IS NOT DISTINCT FROM
- LIKE ... ESCAPE
- NOT LIKE ... ESCAPE
- SIMILAR TO
- NOT SIMILAR TO
- SIMILAR TO ... ESCAPE
- NOT SIMILAR TO ... ESCAPE
Moreover, the two forms of LIKE are handled differently:
- Two-argument LIKE: uses Java regular expression semantics.
- LIKE with ESCAPE: uses SQL % and _ wildcard semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Align YAML Transform BETWEEN, IN, and LIKE semantics with Flink SQL [flink-cdc]
Posted by "haruki-830 (via GitHub)" <gi...@apache.org>.
haruki-830 commented on PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#issuecomment-4979366191
@lvyanquan Could u PTAL when you have time? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#discussion_r3621108186
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##########
@@ -322,6 +335,131 @@ private static Java.Rvalue generateUnaryOperation(
return new Java.UnaryOperation(Location.NOWHERE, operator, atom);
}
+ private static Java.Rvalue generateFunctionOperation(String functionName, Java.Rvalue[] atoms) {
+ return new Java.MethodInvocation(Location.NOWHERE, null, functionName, atoms);
+ }
+
+ private static Java.Rvalue generateLazyBinaryFunctionOperation(
+ Context context, SqlBasicCall sqlBasicCall, String functionName, Java.Rvalue[] atoms) {
+ if (atoms.length != 2) {
+ throw new ParseException("Unrecognized expression: " + sqlBasicCall.toString());
+ }
+ Java.Rvalue rightOperandSupplier =
+ new Java.AmbiguousName(
+ Location.NOWHERE,
+ new String[] {
+ "new java.util.function.Supplier<Boolean>() { public Boolean get() { return "
+ + atoms[1]
+ + "; } }"
+ });
+ return generateFunctionOperation(
+ functionName, new Java.Rvalue[] {atoms[0], rightOperandSupplier});
+ }
+
+ private static Java.Rvalue generateLogicalBinaryOperation(
+ Context context, SqlBasicCall sqlBasicCall, Java.Rvalue[] atoms, boolean isAnd) {
+ if (atoms.length != 2) {
+ throw new ParseException("Unrecognized expression: " + sqlBasicCall.toString());
+ }
+ boolean leftNullable = isExpressionNullable(context, sqlBasicCall.getOperandList().get(0));
+ boolean rightNullable = isExpressionNullable(context, sqlBasicCall.getOperandList().get(1));
+ if (!leftNullable && !rightNullable) {
+ return generateBinaryOperation(context, sqlBasicCall, atoms, isAnd ? "&&" : "||");
+ }
+ if (!leftNullable) {
+ return generateLeftNonNullableLogicalOperation(atoms, isAnd);
+ }
+ // TODO: This nullable-left path still relies on JIT escape analysis to optimize
+ // per-evaluation Supplier allocation. Introduce statement-level codegen
+ // (for example, GeneratedExpression/ScriptEvaluator) to avoid the potential cost.
+ return generateLazyBinaryFunctionOperation(
+ context, sqlBasicCall, isAnd ? "and" : "or", atoms);
+ }
+
+ private static Java.Rvalue generateLeftNonNullableLogicalOperation(
+ Java.Rvalue[] atoms, boolean isAnd) {
+ if (isAnd) {
+ return new Java.ConditionalExpression(
+ Location.NOWHERE,
+ atoms[0],
+ atoms[1],
+ new Java.AmbiguousName(Location.NOWHERE, new String[] {"Boolean.FALSE"}));
+ }
+ return new Java.ConditionalExpression(
+ Location.NOWHERE,
+ atoms[0],
+ new Java.AmbiguousName(Location.NOWHERE, new String[] {"Boolean.TRUE"}),
+ atoms[1]);
+ }
+
+ private static boolean isExpressionNullable(Context context, SqlNode sqlNode) {
+ if (sqlNode instanceof SqlIdentifier) {
+ return isIdentifierNullable(context, (SqlIdentifier) sqlNode);
+ }
+ if (sqlNode instanceof SqlLiteral) {
+ return ((SqlLiteral) sqlNode).getValue() == null;
+ }
+ if (sqlNode instanceof SqlBasicCall) {
+ return isBasicCallNullable(context, (SqlBasicCall) sqlNode);
+ }
+ return true;
+ }
Review Comment:
Instead of deducing `isIdentifierNullable` at the Janino compiler level, can we check the corresponding `RelNode` nullability during Calcite validation?
Current implementation is OK though, as the compilation happens just once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation [flink-cdc]
Posted by "haruki-830 (via GitHub)" <gi...@apache.org>.
haruki-830 commented on code in PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474#discussion_r3621645536
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##########
@@ -322,6 +335,131 @@ private static Java.Rvalue generateUnaryOperation(
return new Java.UnaryOperation(Location.NOWHERE, operator, atom);
}
+ private static Java.Rvalue generateFunctionOperation(String functionName, Java.Rvalue[] atoms) {
+ return new Java.MethodInvocation(Location.NOWHERE, null, functionName, atoms);
+ }
+
+ private static Java.Rvalue generateLazyBinaryFunctionOperation(
+ Context context, SqlBasicCall sqlBasicCall, String functionName, Java.Rvalue[] atoms) {
+ if (atoms.length != 2) {
+ throw new ParseException("Unrecognized expression: " + sqlBasicCall.toString());
+ }
+ Java.Rvalue rightOperandSupplier =
+ new Java.AmbiguousName(
+ Location.NOWHERE,
+ new String[] {
+ "new java.util.function.Supplier<Boolean>() { public Boolean get() { return "
+ + atoms[1]
+ + "; } }"
+ });
+ return generateFunctionOperation(
+ functionName, new Java.Rvalue[] {atoms[0], rightOperandSupplier});
+ }
+
+ private static Java.Rvalue generateLogicalBinaryOperation(
+ Context context, SqlBasicCall sqlBasicCall, Java.Rvalue[] atoms, boolean isAnd) {
+ if (atoms.length != 2) {
+ throw new ParseException("Unrecognized expression: " + sqlBasicCall.toString());
+ }
+ boolean leftNullable = isExpressionNullable(context, sqlBasicCall.getOperandList().get(0));
+ boolean rightNullable = isExpressionNullable(context, sqlBasicCall.getOperandList().get(1));
+ if (!leftNullable && !rightNullable) {
+ return generateBinaryOperation(context, sqlBasicCall, atoms, isAnd ? "&&" : "||");
+ }
+ if (!leftNullable) {
+ return generateLeftNonNullableLogicalOperation(atoms, isAnd);
+ }
+ // TODO: This nullable-left path still relies on JIT escape analysis to optimize
+ // per-evaluation Supplier allocation. Introduce statement-level codegen
+ // (for example, GeneratedExpression/ScriptEvaluator) to avoid the potential cost.
+ return generateLazyBinaryFunctionOperation(
+ context, sqlBasicCall, isAnd ? "and" : "or", atoms);
+ }
+
+ private static Java.Rvalue generateLeftNonNullableLogicalOperation(
+ Java.Rvalue[] atoms, boolean isAnd) {
+ if (isAnd) {
+ return new Java.ConditionalExpression(
+ Location.NOWHERE,
+ atoms[0],
+ atoms[1],
+ new Java.AmbiguousName(Location.NOWHERE, new String[] {"Boolean.FALSE"}));
+ }
+ return new Java.ConditionalExpression(
+ Location.NOWHERE,
+ atoms[0],
+ new Java.AmbiguousName(Location.NOWHERE, new String[] {"Boolean.TRUE"}),
+ atoms[1]);
+ }
+
+ private static boolean isExpressionNullable(Context context, SqlNode sqlNode) {
+ if (sqlNode instanceof SqlIdentifier) {
+ return isIdentifierNullable(context, (SqlIdentifier) sqlNode);
+ }
+ if (sqlNode instanceof SqlLiteral) {
+ return ((SqlLiteral) sqlNode).getValue() == null;
+ }
+ if (sqlNode instanceof SqlBasicCall) {
+ return isBasicCallNullable(context, (SqlBasicCall) sqlNode);
+ }
+ return true;
+ }
Review Comment:
Thanks for the suggestion. I agree that deriving nullability from Calcite/RelNode would be cleaner.
Since the current check only runs once during expression compilation and is conservative for unknown expressions, I kept it local in this PR to avoid expanding the scope. I think it would be better to revisit this in the follow-up PR for statement-level Transform expression codegen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40151][runtime] Support missing YAML Transform predicates and nullable BOOLEAN logical evaluation [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan merged PR #4474:
URL: https://github.com/apache/flink-cdc/pull/4474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org