Re: [I] [Bug] Kafka debezium-json cannot infer primary keys from standard Debezium message key [paimon]

["vbhanuchander-lang (via GitHub)" <gi...@apache.org>]

vbhanuchander-lang commented on issue #8961:
URL: https://github.com/apache/paimon/issues/8961#issuecomment-5247346734

   I traced this through the code and can confirm the diagnosis, including the asymmetry with the Avro path.
   
   `DebeziumJsonRecordParser` inherits `AbstractJsonRecordParser.extractPrimaryKeys()`, which reads only the value:
   
   ```java
   // AbstractJsonRecordParser.java:114
   protected List<String> extractPrimaryKeys() {
       ArrayNode pkNames = getNodeAs(root, primaryField(), ArrayNode.class);
       if (pkNames == null) {
           return Collections.emptyList();
       }
       ...
   ```
   
   while the Avro parser already does the right thing with the Kafka key:
   
   ```java
   // DebeziumAvroRecordParser.java:123
   protected List<String> extractPrimaryKeys() {
       if (keyRecord == null) {
           return Collections.emptyList();
       }
       Schema keySchema = sanitizedSchema(keyRecord.getSchema());
       return keySchema.getFields().stream().map(Schema.Field::name).collect(...);
   }
   ```
   
   So the two Debezium formats disagree on where primary keys come from, and `KafkaDebeziumJsonDeserializationSchema` is already putting the parsed key into `CdcSourceRecord` — the JSON parser simply never looks at it. `currentRecord` is available on `AbstractRecordParser`, so the fallback you propose needs no plumbing changes.
   
   One addition to your suggested behaviour, if it's welcome: the key needs the same schema-enabled / schema-disabled distinction the value already gets via `hasSchema`, since a schema-enabled key nests the fields under `schema.fields[].field` while a schema-disabled key is a flat object. Handling only one shape would fix half the reports.
   
   @TKilome you ticked "willing to submit a PR" — are you still working on this? I don't want to duplicate your effort. If you haven't started, I'm happy to send a PR implementing exactly the four points you listed, with tests for both key shapes and for the `pkNames`-present case so the existing behaviour stays covered. I'll wait a couple of days for a reply before opening anything.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org