You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "goutamadwant (via GitHub)" <gi...@apache.org> on 2026/08/30 19:33:02 UTC

[PR] [Fix][Format] Preserve Avro field case during serialization [seatunnel]

goutamadwant opened a new pull request, #12009:
URL: https://github.com/apache/seatunnel/pull/12009

   ### Purpose of this pull request
   
   Fixes #12008.
   
   `SeaTunnelRowTypeToAvroSchemaConverter` preserves field names when it builds the Avro schema, while `RowToAvroConverter` lowercased names before looking them up in that schema. Avro field lookup is case-sensitive, so mixed-case fields failed before the record could be serialized. This patch uses the schema's exact field names for both top-level and nested `ROW` values.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. Previously, a schema containing a field such as `CustomerID` or a nested field such as `InnerID` failed during Avro serialization. These fields now serialize and deserialize with their original case.
   
   Existing lowercase schemas behave unchanged. This patch does not change public APIs, configuration, dependencies, generated schemas, or the serialized layout of schemas that already worked.
   
   ### How was this patch tested?
   
   Added byte-level Avro serialization/deserialization regression tests for:
   
   - case-distinct top-level fields, `CustomerID` and `customerid`
   - a nested `ROW` field named `InnerID`
   - existing lowercase-field behavior through the complete Avro module test suite
   
   Verification:
   
   - JDK 8: `./mvnw -pl seatunnel-formats/seatunnel-format-avro verify` (6 tests passed)
   - JDK 11: `./mvnw -pl seatunnel-formats/seatunnel-format-avro verify` (6 tests passed)
   - JDK 8 shared-consumer reactor: `./mvnw -q -DskipTests verify -pl seatunnel-formats/seatunnel-format-avro,seatunnel-connectors-v2/connector-kafka,seatunnel-connectors-v2/connector-pulsar -am` (passed)
   - `git diff --check upstream/dev..HEAD` (passed)
   
   ### Check list
   
   * [x] No new Jar binary package is added.
   * [x] Documentation is not required because this restores the existing case-preserving schema behavior; no configuration or documented format contract changes.
   * [x] No incompatible change is introduced, so `incompatible-changes.md` does not need an update.
   * [x] The connector checklist is not applicable because this change is limited to the shared Avro format module.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Format] Preserve Avro field case during serialization [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #12009:
URL: https://github.com/apache/seatunnel/pull/12009


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Format] Preserve Avro field case during serialization [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #12009:
URL: https://github.com/apache/seatunnel/pull/12009#issuecomment-5476926514

   Thanks for the very thorough parallel review, @SEZ9 — this is a great complement to mine.
   
   I went back and directly verified the one open factual question your review raises (Issues 2 and 5): whether `AvroToRowConverter` (the read path) still has any `toLowerCase()` on field names that could make the write-side fix asymmetric. I pulled `AvroToRowConverter.java` at the PR head (`72d861a7`) and there is no `toLowerCase()` anywhere in that file — every lookup is already exact-case:
   
   ```java
   if (record.getSchema().getField(fieldNames[i]) == null) { ... }   // AvroToRowConverter.java:83
   values[i] = convertField(rowType.getFieldType(i), record.get(fieldNames[i]));  // AvroToRowConverter.java:87
   ```
   
   So the read path was never part of the bug and is untouched by this PR, which confirms your own read of the current code ("the fix is correct... as written today") — I just wanted to close the loop on that specific factual check rather than leave it as an open question.
   
   On the rest of your findings: I agree Issues 1 and 3 (bind by `Schema.Field` position instead of by name, so the write side and the generated schema can't structurally drift) are good hardening for the future — today there's no bug there since `recordSchema`/`schema` and the row type are built from the same source in the same method call, but making that invariant structural rather than implicit is a legitimate defensive improvement. Issues 4, 6, 7 and 8 (changelog note, null/nullable-union coverage, and `ARRAY<ROW>`/`MAP<_, ROW>` nested-collection coverage) are all sensible, non-blocking test/doc improvements that would strengthen this further.
   
   None of these are correctness blockers for the fix itself — they're all Medium/Low hardening and coverage suggestions on code paths that are provably correct as written today, which matches your own severity ratings. My merge recommendation from my earlier review stands: **Ready to merge after fixes**, and the only actual blocker is still the pre-existing CI flakes unrelated to this diff (`CoordinatorServiceTest` timeout in `seatunnel-engine-server`, `CouchbaseIT` container-startup flake) — not any code issue. Happy to see the position-binding and extra test-coverage suggestions land either in this PR or as fast, low-risk follow-ups.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org