You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "haruki-830 (via GitHub)" <gi...@apache.org> on 2026/08/04 01:40:12 UTC
[PR] [FLINK-40313][runtime] Support nested collection functions in Transforms [flink-cdc]
haruki-830 opened a new pull request, #4496:
URL: https://github.com/apache/flink-cdc/pull/4496
#### Summary
This commit extends Flink CDC Transform expressions with Flink SQL collection constructors and functions. It adds support for constructing and operating on nested ARRAY, MAP, and ROW values in projection and filter expressions.
#### Key Changes
1. Collection Constructors
- Added support for `ARRAY[...]`, `MAP[...]`, and `ROW(...)`.
- Added recursive type coercion for nested collection constructors.
- Added ROW type conversion with generated field names such as `f0` and `f1`.
2. Collection Functions
- Added support for `CARDINALITY`, `ARRAY_CONTAINS`, `ARRAY_POSITION`, and `ELEMENT`.
- Supports arrays and maps where applicable, including nested values and NULL handling.
- Reports an error when `ELEMENT` receives an array containing multiple elements.
3. Parser and Expression Compilation
- Enabled explicit `ROW(...)` constructors while preserving existing MySQL conformance behavior.
- Added Calcite operator registration and Janino expression generation for the new constructors and functions.
- Supports the new expressions in both projections and filters.
4. Test Coverage and Documentation
- Added unit tests for constructors, functions, NULL behavior, nested values, and error cases.
- Added Transform specification tests for projections, filters, and nested collection types.
- Updated the English and Chinese Transform documentation.
#### JIRA Reference
[https://issues.apache.org/jira/browse/FLINK-40313](https://issues.apache.org/jira/browse/FLINK-40313)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40313][runtime] Support nested collection functions in Transforms [flink-cdc]
Posted by "lvyanquan (via GitHub)" <gi...@apache.org>.
lvyanquan commented on code in PR #4496:
URL: https://github.com/apache/flink-cdc/pull/4496#discussion_r3712478901
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/typeutils/CalciteDataTypeConverter.java:
##########
@@ -339,6 +339,10 @@ public static DataType convertCalciteRelDataTypeToDataType(RelDataType relDataTy
case VARIANT:
return DataTypes.VARIANT();
case ROW:
+ return DataTypes.ROW(
+ relDataType.getFieldList().stream()
+ .map(field -> convertCalciteRelDataTypeToDataType(field.getType()))
+ .toArray(DataType[]::new));
Review Comment:
Could we preserve the original ROW field names here? `DataTypes.ROW(DataType...)` regenerates all field names as `f0`, `f1`, etc. For example, if `complex_row_` is `ROW<name STRING, length INT>`, `ELEMENT(ARRAY[complex_row_])` will incorrectly produce `ROW<f0 STRING, f1 INT>`. The same issue affects named ROW types nested in ARRAY or MAP and may change the downstream schema.
Please construct the ROW type using Calcite’s field names, while normalizing Calcite-generated aliases for explicit `ROW(...)` constructors to `f0`, `f1`, etc. It would also be good to add a regression test covering a named ROW through `ELEMENT`, ARRAY, or MAP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40313][runtime] Support nested collection functions in Transforms [flink-cdc]
Posted by "haruki-830 (via GitHub)" <gi...@apache.org>.
haruki-830 commented on code in PR #4496:
URL: https://github.com/apache/flink-cdc/pull/4496#discussion_r3718907958
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/typeutils/CalciteDataTypeConverter.java:
##########
@@ -339,6 +339,10 @@ public static DataType convertCalciteRelDataTypeToDataType(RelDataType relDataTy
case VARIANT:
return DataTypes.VARIANT();
case ROW:
+ return DataTypes.ROW(
+ relDataType.getFieldList().stream()
+ .map(field -> convertCalciteRelDataTypeToDataType(field.getType()))
+ .toArray(DataType[]::new));
Review Comment:
Fixed in the latest commit. Added a regression test as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40313][runtime] Support nested collection functions in Transforms [flink-cdc]
Posted by "haruki-830 (via GitHub)" <gi...@apache.org>.
haruki-830 commented on code in PR #4496:
URL: https://github.com/apache/flink-cdc/pull/4496#discussion_r3780606115
##########
docs/content/docs/core-concept/transform.md:
##########
@@ -282,6 +282,13 @@ Struct functions are used to access elements in ARRAY, MAP, ROW, and VARIANT typ
| Function | Janino Code | Description |
| -------- | ----------- | ----------- |
+| ARRAY[value, ...] | array(value, ...) | Creates an array from the given values. The values must have a common type. |
Review Comment:
ok^.^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40313][runtime] Support nested collection functions in Transforms [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian merged PR #4496:
URL: https://github.com/apache/flink-cdc/pull/4496
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40313][runtime] Support nested collection functions in Transforms [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #4496:
URL: https://github.com/apache/flink-cdc/pull/4496#discussion_r3776652646
##########
docs/content/docs/core-concept/transform.md:
##########
@@ -282,6 +282,13 @@ Struct functions are used to access elements in ARRAY, MAP, ROW, and VARIANT typ
| Function | Janino Code | Description |
| -------- | ----------- | ----------- |
+| ARRAY[value, ...] | array(value, ...) | Creates an array from the given values. The values must have a common type. |
Review Comment:
Shall we add a bad-case test where element types differ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org