You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Shawn-Hx (via GitHub)" <gi...@apache.org> on 2025/03/06 08:44:41 UTC
[PR] [FLINK-37429][transform] Map each column name to a new name in generated expression [flink-cdc]
Shawn-Hx opened a new pull request, #3939:
URL: https://github.com/apache/flink-cdc/pull/3939
This close [FLINK-37429](https://issues.apache.org/jira/browse/FLINK-37429).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37429][transform] Map each column name to a new name in generated expression [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3939:
URL: https://github.com/apache/flink-cdc/pull/3939#discussion_r1982965060
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java:
##########
@@ -642,4 +654,16 @@ private static boolean hasAsterisk(SqlNode sqlNode) {
return false;
}
}
+
+ public static Map<String, String> generateColumnNameMap(List<String> originalColumnNames) {
+ int i = 0;
+ Map<String, String> columnNameMap = new HashMap<>();
+ for (String columnName : originalColumnNames) {
+ if (!columnNameMap.containsKey(columnName)) {
+ columnNameMap.put(columnName, MAPPED_COLUMN_NAME_PREFIX + i);
+ i++;
+ }
+ }
+ return columnNameMap;
+ }
Review Comment:
Minor concern: though it's easy and simple to use `field\d` as column name, it would produce cryptic error message if Janino throws an exception about generated expressions.
Some options might be:
1. We can leave "legal" names as is without mangling them, and only map names that are not valid Java identifiers.
2. Try to extract valid characters from original column name, and append them as a hint. For example, `invalid-name` could be mapped to `column1_invalidname` (recognizable, at least).
3. Always provide the column name map information when exceptions are thrown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37429][transform] Map each column name to a new name in generated expression [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3939:
URL: https://github.com/apache/flink-cdc/pull/3939#discussion_r1982945203
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformFilter.java:
##########
@@ -41,11 +42,17 @@ public class TransformFilter implements Serializable {
private final String expression;
private final String scriptExpression;
private final List<String> columnNames;
+ private final Map<String, String> columnNameMap;
- public TransformFilter(String expression, String scriptExpression, List<String> columnNames) {
+ public TransformFilter(
+ String expression,
+ String scriptExpression,
+ List<String> columnNames,
+ Map<String, String> columnNameMap) {
Review Comment:
IIUC `TransformFilter` is static and will not be refreshed after schema changes, but `columnNameMap` might be changed by altering columns. Will inserting columns in the middle affect the numbering system?
`ProjectionColumn` looks fine (since projection stuff should be recreated every time when schema changes), but still need to be verified by an IT case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37429][transform] Map each column name to a new name in generated expression [flink-cdc]
Posted by "ruanhang1993 (via GitHub)" <gi...@apache.org>.
ruanhang1993 commented on code in PR #3939:
URL: https://github.com/apache/flink-cdc/pull/3939#discussion_r1986879305
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformFilterProcessor.java:
##########
@@ -96,45 +96,48 @@ public boolean process(
expressionEvaluator.evaluate(generateParams(record, epochTime, opType, meta));
} catch (InvocationTargetException e) {
LOG.error(
- "Table:{} filter:{} execute failed. {}",
+ "Table:{} filter:{} column name map:{} execute failed. {}",
tableInfo.getName(),
transformFilter.getExpression(),
+ transformFilter.getColumnNameMap(),
e);
throw new RuntimeException(e);
+ } catch (IllegalArgumentException e) {
+ throw e;
Review Comment:
Why do we need to catch IllegalArgumentException here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37429][transform] Map each column name to a new name in generated expression [flink-cdc]
Posted by "Shawn-Hx (via GitHub)" <gi...@apache.org>.
Shawn-Hx commented on code in PR #3939:
URL: https://github.com/apache/flink-cdc/pull/3939#discussion_r1986690929
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java:
##########
@@ -642,4 +654,16 @@ private static boolean hasAsterisk(SqlNode sqlNode) {
return false;
}
}
+
+ public static Map<String, String> generateColumnNameMap(List<String> originalColumnNames) {
+ int i = 0;
+ Map<String, String> columnNameMap = new HashMap<>();
+ for (String columnName : originalColumnNames) {
+ if (!columnNameMap.containsKey(columnName)) {
+ columnNameMap.put(columnName, MAPPED_COLUMN_NAME_PREFIX + i);
+ i++;
+ }
+ }
+ return columnNameMap;
+ }
Review Comment:
I prefer option 3.
For option 1, seems it is not very easy to judge a column name is legal or not considering every java keyword is illegal.
For option 2, a corner case is there are two columns named `a-b` and `a.b` and we still cannot recognize them after mapping.
Have added column name map information to the exception log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37429][transform] Map each column name to a new name in generated expression [flink-cdc]
Posted by "Shawn-Hx (via GitHub)" <gi...@apache.org>.
Shawn-Hx commented on code in PR #3939:
URL: https://github.com/apache/flink-cdc/pull/3939#discussion_r1986683512
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformFilter.java:
##########
@@ -41,11 +42,17 @@ public class TransformFilter implements Serializable {
private final String expression;
private final String scriptExpression;
private final List<String> columnNames;
+ private final Map<String, String> columnNameMap;
- public TransformFilter(String expression, String scriptExpression, List<String> columnNames) {
+ public TransformFilter(
+ String expression,
+ String scriptExpression,
+ List<String> columnNames,
+ Map<String, String> columnNameMap) {
Review Comment:
Every `TransformExpressionKey` has its own `columnNameMap`. If the expression script is not changed, the `columnNameMap` will not change. I have added a test for the schema change scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37429][transform] Map each column name to a new name in generated expression [flink-cdc]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on code in PR #3939:
URL: https://github.com/apache/flink-cdc/pull/3939#discussion_r1988318488
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-values/src/main/java/org/apache/flink/cdc/connectors/values/source/TimestampTypeMetadataColumn.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.cdc.connectors.values.source;
+
+import org.apache.flink.cdc.common.source.SupportedMetadataColumn;
+import org.apache.flink.cdc.common.types.DataType;
+import org.apache.flink.cdc.common.types.DataTypes;
+
+import java.util.Map;
+
+/** A {@link SupportedMetadataColumn} for op_ts. */
Review Comment:
```suggestion
/** A {@link SupportedMetadataColumn} for timestamp-type. */
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37429][transform] Map each column name to a new name in generated expression [flink-cdc]
Posted by "Shawn-Hx (via GitHub)" <gi...@apache.org>.
Shawn-Hx commented on code in PR #3939:
URL: https://github.com/apache/flink-cdc/pull/3939#discussion_r1986690929
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java:
##########
@@ -642,4 +654,16 @@ private static boolean hasAsterisk(SqlNode sqlNode) {
return false;
}
}
+
+ public static Map<String, String> generateColumnNameMap(List<String> originalColumnNames) {
+ int i = 0;
+ Map<String, String> columnNameMap = new HashMap<>();
+ for (String columnName : originalColumnNames) {
+ if (!columnNameMap.containsKey(columnName)) {
+ columnNameMap.put(columnName, MAPPED_COLUMN_NAME_PREFIX + i);
+ i++;
+ }
+ }
+ return columnNameMap;
+ }
Review Comment:
I prefer option 3.
For option 1, seems it it not very easy to judge a column name is legal or not considering every java keyword is illegal.
For option 2, a corner case is there are two columns named `a-b` and `a.b` and we still cannot recognize them after mapping.
Have added column name map information to the exception log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37429][transform] Map each column name to a new name in generated expression [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3939:
URL: https://github.com/apache/flink-cdc/pull/3939#discussion_r1983226904
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java:
##########
Review Comment:
Seems this fix should close FLINK-37326, too. Adding a test case to verify this would be nice.
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java:
##########
@@ -642,4 +654,16 @@ private static boolean hasAsterisk(SqlNode sqlNode) {
return false;
}
}
+
+ public static Map<String, String> generateColumnNameMap(List<String> originalColumnNames) {
+ int i = 0;
+ Map<String, String> columnNameMap = new HashMap<>();
+ for (String columnName : originalColumnNames) {
+ if (!columnNameMap.containsKey(columnName)) {
+ columnNameMap.put(columnName, MAPPED_COLUMN_NAME_PREFIX + i);
+ i++;
+ }
+ }
+ return columnNameMap;
+ }
Review Comment:
Minor concern: though it's easy and simple to use `field\d` as column name, it would produce cryptic error message if Janino throws an exception about generated expressions.
Some options might be:
1. We can leave "legal" names as is without mangling them, and only map names that is not valid Java identifiers.
2. Try to extract valid characters from original column name, and append them as a hint. For example, `invalid-name` could be mapped to `column1_invalidname` (recognizable, at least).
3. Always provide the column name map information when exceptions are thrown.
##########
flink-cdc-runtime/src/test/java/org/apache/flink/cdc/runtime/operators/transform/PostTransformOperatorTest.java:
##########
Review Comment:
Maybe we need to test if name mangling approach works well with schema evolution altering columns.
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/transform/TransformFilter.java:
##########
@@ -41,11 +42,17 @@ public class TransformFilter implements Serializable {
private final String expression;
private final String scriptExpression;
private final List<String> columnNames;
+ private final Map<String, String> columnNameMap;
- public TransformFilter(String expression, String scriptExpression, List<String> columnNames) {
+ public TransformFilter(
+ String expression,
+ String scriptExpression,
+ List<String> columnNames,
+ Map<String, String> columnNameMap) {
Review Comment:
IIUC `TransformFilter` is static and will not be refreshed after schema changes, but `columnNameMap` might change by altering columns. Will inserting columns in the middle affect the numbering system?
`ProjectionColumn` looks fine (since projection stuff should be recreated every time when schema changes), but still need to be verified by an IT case.
##########
flink-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/parser/TransformParser.java:
##########
@@ -95,6 +95,8 @@ public class TransformParser {
private static final Logger LOG = LoggerFactory.getLogger(TransformParser.class);
private static final String DEFAULT_SCHEMA = "default_schema";
private static final String DEFAULT_TABLE = "TB";
+ private static final String MAPPED_COLUMN_NAME_PREFIX = "field";
Review Comment:
Just thinking if we can use RegExp-like names (`$1`, `$2`, ...)? Ignore this comment if it doesn't look right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37429][transform] Map each column name to a new name in generated expression [flink-cdc]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang merged PR #3939:
URL: https://github.com/apache/flink-cdc/pull/3939
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org