Re: [PR] [FLINK-29549]- Flink Glue Catalog integration [flink-connector-aws]

["fmorillo7694 (via GitHub)" <gi...@apache.org>]

fmorillo7694 commented on PR #206:
URL: https://github.com/apache/flink-connector-aws/pull/206#issuecomment-5239656985

   ## Real AWS Glue integration test results (design rationale for `originalName`)
   
   The latest head (b1a05d3) includes fixes driven by an integration test run against a **real Glue Data Catalog** (us-west-2), not Moto/Localstack. Posting the evidence here since it settles a design question a reviewer might reasonably raise: *why store lowercase names plus an `originalName` parameter instead of writing mixed-case names directly?*
   
   ### Empirical facts about real Glue (reproducible via the AWS CLI)
   
   1. **Glue lowercases column names on `CreateTable`.** Creating a table with columns `userId` / `EventTime` stores and returns `userid` / `eventtime`. Glue does *not* preserve mixed-case column names, so writing them as-is silently loses case.
   2. **Glue rejects column-level `Parameters` on partition columns** with `400: Parameters not supported for partition columns`. The `originalName` column-parameter mechanism therefore cannot be used for partition keys — their declared case is preserved in an order-preserving *table-level* parameter (`flink.original-partition-keys`) instead.
   
   ### What case loss breaks (observed before the fix, same suite)
   
   - `DESCRIBE` returned `[userid, eventtime, username]` for a table declared `[userId, EventTime, userName]`
   - `SELECT userId FROM t` → *Column 'userId' not found in any table; did you mean 'userid'?*
   - Partition ops failed to match: spec `{eventDate=...}` vs stored keys `[eventdate]`
   - Streaming `INSERT INTO` / `SELECT` against catalog-registered Kinesis tables failed to plan, and JSON field names on the wire would be lowercased (silent data corruption for consumers)
   
   ### Validation on the current head
   
   25-step integration suite against real AWS, all passing:
   
   - **DDL**: mixed-case databases, tables, and columns round-trip exactly (create / show / describe / drop); `ALTER TABLE ... SET` (now implemented via `UpdateTable`) with a case-preservation regression check; partitioned tables with `ADD/SHOW/DROP PARTITION` on mixed-case partition keys
   - **Streaming**: unbounded job writing to and reading from a real Kinesis stream through source/sink tables registered in this catalog; JSON field names verified case-intact on the wire via `GetRecords`; `PARTITIONED BY` on a Kinesis table verified to set the record partition key from the (case-restored) column
   - Unit tests: 126/126
   
   One known behavior worth documenting: partition columns are appended after data columns on read-back (Glue models partition keys outside the storage descriptor, like Hive), so declaring partition columns last keeps the schema order stable across the round-trip.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org