You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "DanielLeens (via GitHub)" <gi...@apache.org> on 2026/08/08 15:58:36 UTC
[PR] [Docs][Connector-V2] Clarify Kafka avro format vs Confluent Schema Registry [seatunnel]
DanielLeens opened a new pull request, #11706:
URL: https://github.com/apache/seatunnel/pull/11706
### Purpose of this pull request
A user asked (via a support-bot Q&A export I was reviewing for documentation gaps) whether
SeaTunnel's Kafka source `avro` format supports Confluent Schema-Registry-encoded Avro messages
("confluent-avro"). The existing FAQ entry lists `avro` as a supported format but doesn't clarify
this, so the question keeps coming up.
Verified against current source: `MessageFormat.AVRO` deserializes via `AvroDeserializationSchema`
(`connector-kafka/.../serialize` and `.../source/KafkaSourceConfig.java`, `case AVRO:`), which has no
schema-registry-header-stripping logic. This is different from `PROTOBUF`, which explicitly supports
it via `strip_schema_registry_header` / `SchemaRegistryAwareProtobufDeserializationSchema` (see the
existing "Protobuf with Schema Registry wire format" section in the same doc). Concretely: if a topic
was produced with Confluent's `KafkaAvroSerializer`, each message carries a 5-byte wire-format prefix
(magic byte + schema id) before the Avro payload. `format = avro` does not strip that prefix, so
deserialization will fail or produce corrupted data.
Extended the existing "What message formats does Kafka Source support?" FAQ entry (`docs/en` and
`docs/zh`) to state this limitation explicitly, at the same level of detail as the existing
Protobuf-with-Schema-Registry section, and cross-linked to it.
### Does this PR introduce _any_ user-facing change?
No functional change. Documentation-only addition.
### How was this patch tested?
Documentation-only change, so `./mvnw spotless:apply` is a no-op (spotless in this repo only formats
`<java>` and `<pom>`, not markdown). The technical claim was verified directly against current `dev`
HEAD source:
- `seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/config/MessageFormat.java`
-- confirms the enum values, including the absence of a distinct Confluent-Avro variant.
- `seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaSourceConfig.java`
-- confirms the `AVRO` case only consults `avro_schema` / catalog schema, with no schema-registry
header handling, versus the `PROTOBUF` case which explicitly branches on
`STRIP_SCHEMA_REGISTRY_HEADER`.
- Confirmed the cross-reference anchor `#protobuf-with-schema-registry-wire-format` matches the
existing `### Protobuf with Schema Registry wire format` heading in both `docs/en` and `docs/zh`.
### Check list
* [x] No new JAR binary package is added.
* [x] English and Chinese documentation both updated.
* [x] No incompatible configuration or API change is introduced (docs-only).
* [x] N/A -- documentation-only change, no test coverage applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs][Connector-V2] Clarify Kafka avro format vs Confluent Schema Registry [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11706:
URL: https://github.com/apache/seatunnel/pull/11706
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org