[PR] [INLONG-12158][SDK] Support SQL reserved keywords as source field names (backtick-quoted columns) [inlong]

["luchunliang (via GitHub)" <gi...@apache.org>]

luchunliang opened a new pull request, #12159:
URL: https://github.com/apache/inlong/pull/12159

   Fixes #12158 
   
   ### Motivation
   
   Description
   When a source data field is named after a SQL reserved keyword (e.g. from, select, where), the Transform SDK fails to look up the field correctly. JSQLParser requires such identifiers to be quoted with backticks (e.g. `from` as from_source), but ColumnParser previously stored the raw output of expr.toString(), which still contained the surrounding backticks. As a result, the SDK ended up looking up `from` (with backticks) instead of the actual field name from, and the lookup returned null.
   
   Background
   The InLong Transform SDK relies on JSQLParser to parse user-provided transformation SQL. If a field name collides with a SQL reserved keyword, it must be quoted with backticks:
   
   sql
   复制
   SELECT `from` AS from_source, itemid AS itemid FROM source
   Without the backticks, JSQLParser raises:
   
   net.sf.jsqlparser.JSQLParserException: Encountered unexpected token: "," ","
       at line 1, column 116.
   Even when backticks are used, ColumnParser previously stored the raw string `from` (including the backticks) as the field name. That caused sourceData.getField(rowIndex, "from") to return null, because the real field name in the source data is from.
   
   Steps to Reproduce
   Configure a source that contains a field named from (a SQL reserved keyword).
   Use the following transform SQL:
   sql
   复制
   SELECT `from` AS from_source, itemid AS itemid FROM source
   Run the transform.
   Expected Behavior
   The surrounding backticks should be stripped during parsing, and the SDK should use the unquoted field name (from) to look up the value from the source data.
   
   Actual Behavior (before the fix)
   ColumnParser stored the field name as `from` (with backticks), so the field lookup always returned null.
   
   Proposed Fix
   Strip the surrounding backticks in ColumnParser before storing the field name.
   
   File: inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/ColumnParser.java
   
   java
   复制
   public ColumnParser(Column expr) {
       this.fieldName = this.formatFieldName(expr.toString());
   }
   
   public ColumnParser(Function expr) {
       this.fieldName = this.formatFieldName(expr.toString());
   }
   
   private String formatFieldName(String rawFieldName) {
       String tmpFieldName = rawFieldName;
       if (tmpFieldName.startsWith("`")) {
           tmpFieldName = tmpFieldName.substring(1);
       }
       if (tmpFieldName.endsWith("`")) {
           tmpFieldName = tmpFieldName.substring(0, tmpFieldName.length() - 1);
       }
       return tmpFieldName;
   }
   The same backtick-stripping logic is also applied to select-item aliases in TransformProcessor:
   
   File: inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/TransformProcessor.java
   
   java
   复制
   fieldName = exprItem.getAlias().getName();
   // Strip surrounding backticks if present
   if (fieldName != null && fieldName.length() >= 2
           && fieldName.startsWith("`") && fieldName.endsWith("`")) {
       fieldName = fieldName.substring(1, fieldName.length() - 1);
   }
   Affected Components
   inlong-sdk/transform-sdk
   ColumnParser.java — added formatFieldName() to strip backticks
   TransformProcessor.java — added backtick stripping for select-item aliases
   Test Coverage
   Updated a test case in TestCsv2KvProcessor.java to cover a backtick-quoted column name:
   
   java
   复制
   // In TestCsv2KvProcessor.java
   + ",`from` as random_seq"
   Notes
   formatFieldName() currently only handles backticks (`). Other quoting styles such as double quotes ("from") or square brackets ([from]) are not stripped. This can be addressed in a follow-up improvement if needed.
   ### Modifications
   
   <!--Describe the modifications you've done.-->
   
   ### Verifying this change
   
   *(Please pick either of the following options)*
   
   - [ ] This change is a trivial rework/code cleanup without any test coverage.
   
   - [ ] This change is already covered by existing tests, such as:
     *(please describe tests)*
   
   - [ ] This change added tests and can be verified as follows:
   
     *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (10MB)*
     - *Extended integration test for recovery after broker failure*
   
   ### Documentation
   
     - Does this pull request introduce a new feature? (yes / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
     - If a feature is not applicable for documentation, explain why?
     - If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [INLONG-12158][SDK] Support SQL reserved keywords as source field names (backtick-quoted columns) [inlong]

["luchunliang (via GitHub)" <gi...@apache.org>]

luchunliang merged PR #12159:
URL: https://github.com/apache/inlong/pull/12159


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org