You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "DanielLeens (via GitHub)" <gi...@apache.org> on 2026/08/08 15:58:36 UTC

[PR] [Docs][Connector-V2] Clarify Kafka avro format vs Confluent Schema Registry [seatunnel]

DanielLeens opened a new pull request, #11706:
URL: https://github.com/apache/seatunnel/pull/11706

   ### Purpose of this pull request
   
   A user asked (via a support-bot Q&A export I was reviewing for documentation gaps) whether
   SeaTunnel's Kafka source `avro` format supports Confluent Schema-Registry-encoded Avro messages
   ("confluent-avro"). The existing FAQ entry lists `avro` as a supported format but doesn't clarify
   this, so the question keeps coming up.
   
   Verified against current source: `MessageFormat.AVRO` deserializes via `AvroDeserializationSchema`
   (`connector-kafka/.../serialize` and `.../source/KafkaSourceConfig.java`, `case AVRO:`), which has no
   schema-registry-header-stripping logic. This is different from `PROTOBUF`, which explicitly supports
   it via `strip_schema_registry_header` / `SchemaRegistryAwareProtobufDeserializationSchema` (see the
   existing "Protobuf with Schema Registry wire format" section in the same doc). Concretely: if a topic
   was produced with Confluent's `KafkaAvroSerializer`, each message carries a 5-byte wire-format prefix
   (magic byte + schema id) before the Avro payload. `format = avro` does not strip that prefix, so
   deserialization will fail or produce corrupted data.
   
   Extended the existing "What message formats does Kafka Source support?" FAQ entry (`docs/en` and
   `docs/zh`) to state this limitation explicitly, at the same level of detail as the existing
   Protobuf-with-Schema-Registry section, and cross-linked to it.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No functional change. Documentation-only addition.
   
   ### How was this patch tested?
   
   Documentation-only change, so `./mvnw spotless:apply` is a no-op (spotless in this repo only formats
   `<java>` and `<pom>`, not markdown). The technical claim was verified directly against current `dev`
   HEAD source:
   - `seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/config/MessageFormat.java`
     -- confirms the enum values, including the absence of a distinct Confluent-Avro variant.
   - `seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaSourceConfig.java`
     -- confirms the `AVRO` case only consults `avro_schema` / catalog schema, with no schema-registry
     header handling, versus the `PROTOBUF` case which explicitly branches on
     `STRIP_SCHEMA_REGISTRY_HEADER`.
   - Confirmed the cross-reference anchor `#protobuf-with-schema-registry-wire-format` matches the
     existing `### Protobuf with Schema Registry wire format` heading in both `docs/en` and `docs/zh`.
   
   ### Check list
   
   * [x] No new JAR binary package is added.
   * [x] English and Chinese documentation both updated.
   * [x] No incompatible configuration or API change is introduced (docs-only).
   * [x] N/A -- documentation-only change, no test coverage applicable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Docs][Connector-V2] Clarify Kafka avro format vs Confluent Schema Registry [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11706:
URL: https://github.com/apache/seatunnel/pull/11706


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org