You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "thehkkim (via GitHub)" <gi...@apache.org> on 2026/07/21 14:05:53 UTC

[PR] [Feature][Connector-V2][Kafka] Support reading Kafka message headers in source connector [seatunnel]

thehkkim opened a new pull request, #11521:
URL: https://github.com/apache/seatunnel/pull/11521

   ### Purpose of this pull request
   
   This PR adds `kafka_headers_fields` option to the **Kafka source connector**, allowing users to extract Kafka message header values as row fields.
   
   This is the source-side counterpart of [#10335](https://github.com/apache/seatunnel/pull/10335), which added `kafka_headers_fields` to the Kafka sink connector (merged into `dev`, not yet released). With both features in place, Kafka message headers can now be written via the sink and read back via the source, enabling end-to-end header propagation between topics.
   
   ### Does this PR introduce _any_ user-facing change?
   
   A new optional parameter `kafka_headers_fields` is added to the Kafka source connector.
   
   **Previous behavior:** Kafka message headers were only accessible when using `format = NATIVE`, which exposes all headers as a `Map<String, String>` field. For other formats (JSON, TEXT, etc.), headers were silently discarded.
   
   **New behavior:** Users can specify a list of header key names via `kafka_headers_fields`. Each header value is read as a `STRING` field and appended to the output row after the regular schema fields. If a header key is absent in a record, the corresponding field value is `null`.
   
   **Example:**
   
   ```hocon
   source {
     Kafka {
       bootstrap.servers = "localhost:9092"
       topic = "my-topic"
       kafka_headers_fields = ["correlation_id", "trace_id"]
       schema = {
         fields {
           user_id = "int"
           name    = "string"
         }
       }
       format = json
     }
   }
   ```
   
   Output row schema: `user_id` (int), `name` (string), `correlation_id` (string), `trace_id` (string).
   
   > Note: this is a user-facing change relative to the unreleased `dev` branch. The sink-side `kafka_headers_fields` (#10335) is also only available in `dev`.
   
   **Constraints (same as sink):**
   - Cannot be used with `format = NATIVE` (NATIVE already exposes all headers as a Map)
   
   ### How was this patch tested?
   
   **Unit tests** (`connector-kafka`):
   - `KafkaHeadersDeserializationSchemaTest` (new): 7 cases covering header value appending, null header value, missing header key, options/tableId/rowKind preservation, multi-header fields, and collector path
   - `KafkaRecordEmitterTest` (extended): 2 new cases verifying header field injection through the `KafkaEventTime(KafkaHeaders(base))` wrapper chain, and that EVENT_TIME metadata is preserved alongside headers
   - `KafkaSourceConfigTest` (extended): 3 new cases verifying schema extension with header fields, correct deserialization schema wrapping order, and that NATIVE format is unaffected
   
   **E2E test** (`connector-kafka-e2e`):
   - `KafkaIT.testSourceKafkaWithHeaders`: produces 10 records with JSON value and two headers (`correlation_id`, `trace_id`) to a Kafka topic, runs the SeaTunnel job, and asserts via the Assert sink that all four fields (`user_id`, `name`, `correlation_id`, `trace_id`) are non-null for every row
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. [x] Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Kafka] Support reading Kafka message headers in source connector [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11521:
URL: https://github.com/apache/seatunnel/pull/11521


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Kafka] Support reading Kafka message headers in source connector [seatunnel]

Posted by "thehkkim (via GitHub)" <gi...@apache.org>.
thehkkim commented on PR #11521:
URL: https://github.com/apache/seatunnel/pull/11521#issuecomment-5054955515

   @DanielLeens 
   Thank you again for the thorough review — it really helped catch a gap I had missed.              
   
   I went with implementing header injection for the COMPATIBLE_KAFKA_CONNECT_JSON path rather than rejecting it. Here's what changed:
   
   - `ConsumerMetadata`: Added a kafkaHeaderFields field. When `format = compatible_kafka_connect_json` and `kafka_headers_fields` is configured, the header field names are stored here for the emitter to use at runtime.
   - `KafkaSourceConfig`: The catalog table is now extended with header fields for                     
   `COMPATIBLE_KAFKA_CONNECT_JSON` the same way as other formats. The `CompatibleKafkaConnectDeserializationSchema` itself is still initialized with the base catalog table (before extension), since it handles deserialization independently via deserialize(ConsumerRecord, Collector).
   - `KafkaRecordEmitter`: For the `CompatibleKafkaConnectDeserializationSchema` dispatch path, the emitter now checks whether `kafkaHeaderFields` is set on ConsumerMetadata. If so, it wraps the output Collector with a headerInjectingCollector that appends header values to each emitted row after deserialization, keeping the runtime row width consistent with the declared output schema.
   - `KafkaSourceConfigTest`: Updated the test to verify that `COMPATIBLE_KAFKA_CONNECT_JSON` + `kafka_headers_fields` correctly extends the output schema and stores the header field names inConsumerMetadata.
   
   Regarding the build failures — as you pointed out, they are in `connector-starrocks` (StarRocksStreamLoadVisitorTest) and `connector-cdc-base`, neither of which is touched by this PR.      
   
   Please take another look when you get a chance!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org