You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "tianfy (Jira)" <ji...@apache.org> on 2026/02/27 03:19:00 UTC
[jira] [Updated] (FLINK-39171) Oracle pipeline transform fails with quoted uppercase projection columns
[ https://issues.apache.org/jira/browse/FLINK-39171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
tianfy updated FLINK-39171:
---------------------------
Description:
In Oracle CDC {*}pipeline mode{*}, transform projection can fail when projection uses quoted uppercase source columns (for example {{{}`ID`{}}}), with:
{{Column 'ID' not found in any table}}
Observed behavior suggests Oracle pipeline schema columns are normalized to lowercase internally (for example {{{}id{}}}), while transform SQL validation treats quoted identifiers as exact/case-sensitive, causing mismatch.
h3. Reproduction
# Create Oracle table (unquoted column names, Oracle stores as uppercase):
** {{TESTUSER.ORDERS(ID, PRODUCT_ID, QUANTITY, UPDATED_AT)}}
# Run Oracle pipeline with transform projection using quoted uppercase source columns, e.g.:
pipeline:
schema.change.behavior: LENIENT
transform:
- source-table: ORCL19.TESTUSER.ORDERS
primary-keys: id
projection: "`ID` as `id`,`PRODUCT_ID` as `product_id`,`QUANTITY` as `quantity`,`UPDATED_AT` as `updated_at`"
source:
type: oracle
database: ORCL19
tables: ORCL19.TESTUSER.ORDERS
scan.startup.mode: initial
3. Start the job.
h3. Actual behavior
Job fails during post-transform schema handling:
* org.apache.flink.cdc.runtime.operators.transform.exceptions.TransformException
* Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Column 'ID' not found in any table
Stack path includes: PostTransformOperator -> TransformParser.generateProjectionColumns -> Calcite validator
h3. Expected behavior
Oracle pipeline should use a consistent column-name strategy between schema and transform projection:
# Either preserve Oracle column case consistently, or
# Normalize both schema and projection consistently, or
# Resolve quoted identifiers safely for Oracle semantics.
At minimum, projection generated from source metadata should not fail with column-not-found because of internal case normalization mismatch.
h3. Impact
* Oracle pipeline job fails at startup for common projection patterns using original DB column names.
* Blocks end-to-end sync in transform-enabled pipelines.
h3. Workaround
* Convert Oracle source field names in projection to lowercase before submission (for example {{id}} instead of {{ID}}).
* Avoid quoted uppercase source identifiers in transform projection.
h3. Additional notes
If needed, I can provide a minimal runnable reproducer (DDL + full pipeline YAML + complete stack trace).
was:
In Oracle CDC *pipeline mode*, transform projection can fail when projection uses quoted uppercase source columns (for example {{`ID`}}), with:
{{Column 'ID' not found in any table}}
Observed behavior suggests Oracle pipeline schema columns are normalized to lowercase internally (for example {{id}}), while transform SQL validation treats quoted identifiers as exact/case-sensitive, causing mismatch.
This does not reproduce in our non-pipeline Oracle CDC source path.
h3. Reproduction
# Create Oracle table (unquoted column names, Oracle stores as uppercase):
** {{TESTUSER.ORDERS(ID, PRODUCT_ID, QUANTITY, UPDATED_AT)}}
# Run Oracle pipeline with transform projection using quoted uppercase source columns, e.g.:
pipeline:
schema.change.behavior: LENIENT
transform:
- source-table: ORCL19.TESTUSER.ORDERS
primary-keys: id
projection: "`ID` as `id`,`PRODUCT_ID` as `product_id`,`QUANTITY` as `quantity`,`UPDATED_AT` as `updated_at`"
source:
type: oracle
database: ORCL19
tables: ORCL19.TESTUSER.ORDERS
scan.startup.mode: initial
3. Start the job.
> Oracle pipeline transform fails with quoted uppercase projection columns
> ------------------------------------------------------------------------
>
> Key: FLINK-39171
> URL: https://issues.apache.org/jira/browse/FLINK-39171
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.6.0
> Environment: * Flink: 1.20.1
> * Oracle: 19c (Non-CDB,)
> * Startup mode: {{initial}}
> * Flink CDC：3.6.0
> Reporter: tianfy
> Priority: Major
>
> In Oracle CDC {*}pipeline mode{*}, transform projection can fail when projection uses quoted uppercase source columns (for example {{{}`ID`{}}}), with:
> {{Column 'ID' not found in any table}}
> Observed behavior suggests Oracle pipeline schema columns are normalized to lowercase internally (for example {{{}id{}}}), while transform SQL validation treats quoted identifiers as exact/case-sensitive, causing mismatch.
> h3. Reproduction
> # Create Oracle table (unquoted column names, Oracle stores as uppercase):
> ** {{TESTUSER.ORDERS(ID, PRODUCT_ID, QUANTITY, UPDATED_AT)}}
> # Run Oracle pipeline with transform projection using quoted uppercase source columns, e.g.:
> pipeline:
> schema.change.behavior: LENIENT
>
> transform:
> - source-table: ORCL19.TESTUSER.ORDERS
> primary-keys: id
> projection: "`ID` as `id`,`PRODUCT_ID` as `product_id`,`QUANTITY` as `quantity`,`UPDATED_AT` as `updated_at`"
>
> source:
> type: oracle
> database: ORCL19
> tables: ORCL19.TESTUSER.ORDERS
> scan.startup.mode: initial
>
> 3. Start the job.
> h3. Actual behavior
> Job fails during post-transform schema handling:
> * org.apache.flink.cdc.runtime.operators.transform.exceptions.TransformException
> * Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Column 'ID' not found in any table
> Stack path includes: PostTransformOperator -> TransformParser.generateProjectionColumns -> Calcite validator
> h3. Expected behavior
> Oracle pipeline should use a consistent column-name strategy between schema and transform projection:
> # Either preserve Oracle column case consistently, or
> # Normalize both schema and projection consistently, or
> # Resolve quoted identifiers safely for Oracle semantics.
> At minimum, projection generated from source metadata should not fail with column-not-found because of internal case normalization mismatch.
> h3. Impact
> * Oracle pipeline job fails at startup for common projection patterns using original DB column names.
> * Blocks end-to-end sync in transform-enabled pipelines.
> h3. Workaround
> * Convert Oracle source field names in projection to lowercase before submission (for example {{id}} instead of {{ID}}).
> * Avoid quoted uppercase source identifiers in transform projection.
> h3. Additional notes
> If needed, I can provide a minimal runnable reproducer (DDL + full pipeline YAML + complete stack trace).
>
>
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)