You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "haruki-830 (via GitHub)" <gi...@apache.org> on 2026/07/28 05:08:27 UTC
[PR] [FLINK-40240][runtime] Support TRY_CAST, IFNULL and NULLIF in YAML Transform [flink-cdc]
haruki-830 opened a new pull request, #4492:
URL: https://github.com/apache/flink-cdc/pull/4492
#### Summary
This commit adds `TRY_CAST`, `IFNULL`, and `NULLIF` support to YAML Transform projections and filters. It aligns their behavior with Flink SQL while preserving the existing `CAST` failure semantics.
#### Key Changes
1. Function and Syntax Support
- Added operator and type metadata for `TRY_CAST`, `IFNULL`, and `NULLIF`.
- Added preprocessing for `TRY_CAST(expr AS type)` because Calcite 1.32 does not support this syntax directly.
- Handles nested expressions, different letter cases, quoted text, identifiers, and SQL comments.
2. Code Generation and Runtime Semantics
- Added Janino code generation for all three functions.
- `TRY_CAST` supports the same target types as the existing `CAST` implementation and returns `NULL` only for data conversion failures.
- Unsupported conversion paths fail during pipeline validation, while invalid configuration, UDF, and internal errors continue to fail the task.
- Added null-safe `IFNULL` and numeric-aware/deep-equality `NULLIF` evaluation without changing existing `CAST` behavior.
3. Test Coverage and Documentation
- Added parser, code generation, runtime, and YAML specification coverage for projections and filters.
- Covered successful and failed casts, nested expressions, nullability, invalid syntax, and function combinations.
- Verified the changes with both default Flink 1.20 and the Flink 2 profile.
- Updated the English and Chinese Transform documentation.
#### JIRA Reference
[https://issues.apache.org/jira/browse/FLINK-40240](https://issues.apache.org/jira/browse/FLINK-40240)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40240][runtime] Support TRY_CAST, IFNULL and NULLIF in YAML Transform [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #4492:
URL: https://github.com/apache/flink-cdc/pull/4492#issuecomment-5215547400
Could you please rebase with latest master and resolve conflicts if any?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40240][runtime] Support TRY_CAST, IFNULL and NULLIF in YAML Transform [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on PR #4492:
URL: https://github.com/apache/flink-cdc/pull/4492#issuecomment-5252708047
Thanks @haruki-830 for the quick response!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40240][runtime] Support TRY_CAST, IFNULL and NULLIF in YAML Transform [flink-cdc]
Posted by "haruki-830 (via GitHub)" <gi...@apache.org>.
haruki-830 commented on code in PR #4492:
URL: https://github.com/apache/flink-cdc/pull/4492#discussion_r3755830185
##########
docs/content.zh/docs/core-concept/transform.md:
##########
@@ -244,11 +244,15 @@ Flink CDC 使用 [Calcite](https://calcite.apache.org/) 来解析表达式并且
| CASE WHEN condition1 THEN result1 (WHEN condition2 THEN result2)* (ELSE result_z) END | 嵌套三元表达式 | 当第一个 conditionX 满足时，返回 resultX。如果没有条件满足，如果提供了 result_z 则返回 result_z，否则返回 NULL。 |
| COALESCE(value1 [, value2]*) | coalesce(Object... objects) | 返回第一个不为 NULL 的参数。如果所有参数都为 NULL，则返回 NULL。返回类型是所有参数中限制最少的公共类型。如果所有参数都可为空，则返回类型也可为空。 |
| IF(condition, true_value, false_value) | condition ? true_value : false_value | 如果条件满足，返回 true_value，否则返回 false_value。例如，IF(5 > 3, 5, 3) 返回 5。 |
+| IFNULL(value, replacement) | ifNull(value, replacement) | 当 value 为 NULL 时返回 replacement，否则返回 value。两个参数必须存在公共类型。仅当 replacement 可为 NULL 时，结果才可为 NULL。 |
+| NULLIF(value1, value2) | nullIf(value1, value2) | 当 value1 与 value2 相等时返回 NULL，否则返回 value1。返回类型为 value1 对应的可空类型。 |
Review Comment:
Thanks for pointing this out! [FLINK-40203](https://issues.apache.org/jira/browse/FLINK-40203) / PR [#4484](https://github.com/apache/flink-cdc/pull/4484) is introducing an opt-in `FLINK_SQL` expression semantics mode, under which `=` compares numeric operands across numeric types and is therefore consistent with `NULLIF`. The default mode intentionally preserves the existing type-sensitive `=` behavior for backward compatibility. In the latest commit, I have documented this difference rather than duplicating the comparison-semantics changes from FLINK-40203.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40240][runtime] Support TRY_CAST, IFNULL and NULLIF in YAML Transform [flink-cdc]
Posted by "haruki-830 (via GitHub)" <gi...@apache.org>.
haruki-830 commented on PR #4492:
URL: https://github.com/apache/flink-cdc/pull/4492#issuecomment-5101151730
@lvyanquan @yuxiqian PTAL, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40240][runtime] Support TRY_CAST, IFNULL and NULLIF in YAML Transform [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on code in PR #4492:
URL: https://github.com/apache/flink-cdc/pull/4492#discussion_r3711713084
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/functions/impl/LogicalFunctions.java:
##########
@@ -102,4 +104,45 @@ public static Object coalesce(Object... objects) {
}
return null;
}
+
+ public static <T> T ifNull(T value, T replacement) {
+ return value != null ? value : replacement;
+ }
+
+ public static <T> T nullIf(T value, Object comparison) {
+ return valuesEqualForNullIf(value, comparison) ? null : value;
+ }
+
+ private static boolean valuesEqualForNullIf(Object value, Object comparison) {
+ if (value == null || comparison == null) {
+ return false;
+ }
+ if (!(value instanceof Number) || !(comparison instanceof Number)) {
+ return Objects.deepEquals(value, comparison);
+ }
+
+ Number left = (Number) value;
+ Number right = (Number) comparison;
+ if (left instanceof Double || right instanceof Double) {
+ double leftValue = left.doubleValue();
+ double rightValue = right.doubleValue();
+ if (Double.isFinite(leftValue) && Double.isFinite(rightValue)) {
+ return BigDecimal.valueOf(leftValue).compareTo(BigDecimal.valueOf(rightValue)) == 0;
+ }
+ return Double.compare(leftValue, rightValue) == 0;
+ }
+ if (left instanceof Float || right instanceof Float) {
+ return Float.compare(left.floatValue(), right.floatValue()) == 0;
Review Comment:
Float.compare and Double.compare do not match SQL numeric equality semantics. In particular, they treat -0.0 and 0.0 as different and treat two NaN values as equal. As a result, NULLIF(-0.0, 0.0) incorrectly returns the first value, while NULLIF(NaN, NaN) incorrectly returns NULL.
Please compare floating-point operands using primitive numeric equality after coercing them to the common numeric type, rather than using Float.compare or Double.compare.
Tests covering signed zero, NaN, and infinity should also be added.
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/functions/impl/LogicalFunctions.java:
##########
@@ -102,4 +104,45 @@ public static Object coalesce(Object... objects) {
}
return null;
}
+
+ public static <T> T ifNull(T value, T replacement) {
+ return value != null ? value : replacement;
+ }
+
+ public static <T> T nullIf(T value, Object comparison) {
+ return valuesEqualForNullIf(value, comparison) ? null : value;
+ }
+
+ private static boolean valuesEqualForNullIf(Object value, Object comparison) {
+ if (value == null || comparison == null) {
+ return false;
+ }
+ if (!(value instanceof Number) || !(comparison instanceof Number)) {
+ return Objects.deepEquals(value, comparison);
+ }
+
+ Number left = (Number) value;
+ Number right = (Number) comparison;
+ if (left instanceof Double || right instanceof Double) {
+ double leftValue = left.doubleValue();
+ double rightValue = right.doubleValue();
+ if (Double.isFinite(leftValue) && Double.isFinite(rightValue)) {
+ return BigDecimal.valueOf(leftValue).compareTo(BigDecimal.valueOf(rightValue)) == 0;
+ }
+ return Double.compare(leftValue, rightValue) == 0;
+ }
+ if (left instanceof Float || right instanceof Float) {
+ return Float.compare(left.floatValue(), right.floatValue()) == 0;
Review Comment:
The Float branch in `valuesEqualForNullIf` downgrades the other side via `floatValue()`, so `nullIf(16_777_217L, 16_777_217f)` wrongly returns null — beyond float's 24-bit mantissa (2²⁴), values get rounded and compared equal, contradicting both SQL semantics (which promotes to DOUBLE) and the `=` operator here. Since `float → double` is lossless, please promote to double like the Double branch above does. A regression test for `NULLIF(BIGINT, FLOAT)` with values beyond 2²⁴ would be appreciated.
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/functions/impl/CastingFunctions.java:
##########
@@ -264,7 +264,51 @@ public static BigDecimal castToBigDecimal(Object object, int precision, int scal
}
public static LocalDateTime castToTimestamp(Object object, String timezone) {
- ZoneId zoneId = ZoneId.of(timezone);
+ return castToTimestamp(object, ZoneId.of(timezone), false);
+ }
+
+ public static String tryCastToString(Object object) {
+ return castToString(object);
+ }
+
+ public static Boolean tryCastToBoolean(Object object) {
+ return castToBoolean(object);
+ }
+
+ public static Byte tryCastToByte(Object object) {
+ return castToByte(object);
+ }
+
+ public static Short tryCastToShort(Object object) {
+ return castToShort(object);
+ }
+
+ public static Integer tryCastToInteger(Object object) {
+ return castToInteger(object);
+ }
+
+ public static Long tryCastToLong(Object object) {
+ return castToLong(object);
+ }
+
+ public static Float tryCastToFloat(Object object) {
+ return castToFloat(object);
+ }
+
+ public static Double tryCastToDouble(Object object) {
+ return castToDouble(object);
+ }
+
+ public static BigDecimal tryCastToBigDecimal(Object object, int precision, int scale) {
Review Comment:
Except for TIMESTAMP, all newly added tryCastToXxx methods simply delegate to the existing castToXxx implementations. Therefore, TRY_CAST cannot reliably detect conversion failures or distinguish its behavior from CAST.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40240][runtime] Support TRY_CAST, IFNULL and NULLIF in YAML Transform [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on code in PR #4492:
URL: https://github.com/apache/flink-cdc/pull/4492#discussion_r3756719729
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/JaninoCompiler.java:
##########
@@ -564,7 +569,9 @@ private static boolean isIdentifierNullable(Context context, SqlIdentifier sqlId
}
private static boolean isBasicCallNullable(Context context, SqlBasicCall sqlBasicCall) {
- if (sqlBasicCall.getOperator().getName().equalsIgnoreCase("IFNULL")) {
+ if (sqlBasicCall.getOperator().getName().equalsIgnoreCase("IFNULL")
Review Comment:
Thanks for the fix — the precedence behavior is now correct and well tested.
One structural concern though: the current approach guards each built-in branch with a negative check (`!findUserDefinedFunction(...).isPresent()`), scattered across four places (the `TRY_CAST` entry, `case NULLIF:` in the SqlKind switch, the IFNULL/NULLIF branches here, and `isBasicCallNullable`). This inverts the established convention in this method — the `orElseGet` fallback below is structurally "UDF first, built-in fallback", whereas the fix is "built-in first, exclude UDFs case by case".
Would you consider consolidating this into a single positive dispatch point at the top of function-call translation? Something like:
```java
Optional<UserDefinedFunctionDescriptor> udf = findUserDefinedFunction(context, functionName);
if (udf.isPresent()) {
return generateUdfInvocation(context, udf.get(), atoms);
}
// built-in special cases (TRY_CAST / IF / IFNULL / NULLIF / ...) below
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40240][runtime] Support TRY_CAST, IFNULL and NULLIF in YAML Transform [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan merged PR #4492:
URL: https://github.com/apache/flink-cdc/pull/4492
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40240][runtime] Support TRY_CAST, IFNULL and NULLIF in YAML Transform [flink-cdc]
Posted by "haruki-830 (via GitHub)" <gi...@apache.org>.
haruki-830 commented on PR #4492:
URL: https://github.com/apache/flink-cdc/pull/4492#issuecomment-5189881799
@lvyanquan Thanks for the careful review! I’ve addressed the requested changes in the latest commit. The branch has also been rebased onto the latest master. Please take another look when you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org