You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "thehkkim (via GitHub)" <gi...@apache.org> on 2026/01/13 23:08:13 UTC
[PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
thehkkim opened a new pull request, #10335:
URL: https://github.com/apache/seatunnel/pull/10335
### Purpose of this pull request
This PR adds support for Kafka message headers in the Kafka sink connector through a new `kafka_headers_fields` configuration option.
Users can now specify which fields should be set as Kafka message headers, enabling better message routing, filtering, and metadata management in Kafka consumers without parsing the message body.
### Does this PR introduce _any_ user-facing change?
Yes. This PR introduces a new configuration option `kafka_headers_fields` for the Kafka sink connector.
**New Feature:**
- Added `kafka_headers_fields` configuration option to specify which fields should be added as Kafka message headers
- Fields specified as headers are automatically excluded from the message value to avoid data duplication
**Usage Example:**
Configuration:
```hocon
sink {
kafka {
topic = "test_topic"
bootstrap.servers = "localhost:9092"
format = json
partition_key_fields = ["id"]
kafka_headers_fields = ["source", "traceId"] # New option
}
}
```
Input data:
```json
{"id": 1, "name": "test", "source": "web", "traceId": "trace-123"}
```
Output to Kafka:
- **Headers**: `source=web`, `traceId=trace-123`
- **Key**: `{"id": 1}`
- **Value**: `{"id": 1, "name": "test"}` (header fields excluded)
**Benefits:**
- Enables header-based routing and filtering in Kafka consumers
- Supports distributed tracing by passing trace IDs via headers
- Allows metadata tagging without parsing message body
- Follows standard Kafka pattern where headers contain metadata
**Backward Compatibility:**
- Fully backward compatible - new optional configuration
- Existing configurations continue to work without any changes
### How was this patch tested?
**1. Unit Tests:**
Added comprehensive test coverage in `DefaultSeaTunnelRowSerializerTest`:
- `testKafkaHeaders()` - Verifies fields are correctly added to Kafka message headers
- `testKafkaHeadersWithNullValue()` - Ensures null values are properly handled (not added to headers)
- `testHeaderFieldsExcludedFromValue()` - Confirms header fields are excluded from message value
**2. Test Results:**
```
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
```
All tests pass successfully, verifying:
- ✅ Headers are correctly populated with specified field values
- ✅ Header fields are automatically excluded from the serialized value
- ✅ Null values are handled properly (skipped in headers)
- ✅ Multiple header fields work correctly
- ✅ Field values are converted to strings as header values
**3. Test Coverage:**
- Tested with JSON format (primary use case)
- Verified serialization logic for both headers and values
- Confirmed backward compatibility when `kafka_headers_fields` is not configured
**4. Code Quality:**
- ✅ Passed Apache SeaTunnel Spotless formatting checks
- ✅ Follows existing code patterns in the Kafka connector
- ✅ Consistent with `partition_key_fields` implementation pattern
### Check list
* [x] Code changes are covered with comprehensive unit tests
* [x] Documentation updated to describe the new feature
- Updated `docs/en/connectors/sink/Kafka.md` with feature description and usage example
- Updated `docs/zh/connectors/sink/Kafka.md` with Chinese translation
* [x] Added configuration examples in documentation
* [ ] No new Jar binary packages added
* [ ] Not a new connector (enhancement to existing Kafka connector)
* [x] Code follows Apache SeaTunnel coding standards and passes all checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on code in PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#discussion_r2837001508
##########
seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/serialize/DefaultSeaTunnelRowSerializer.java:
##########
@@ -288,6 +378,21 @@ private static Function<SeaTunnelRow, SeaTunnelRow> createKeyRowExtractor(
};
}
+ private static Function<SeaTunnelRow, SeaTunnelRow> createValueRowExtractor(
+ SeaTunnelRowType valueType, List<String> headerFieldNames, SeaTunnelRowType rowType) {
+ int[] valueIndex = new int[valueType.getTotalFields()];
+ for (int i = 0; i < valueType.getTotalFields(); i++) {
+ valueIndex[i] = rowType.indexOf(valueType.getFieldName(i));
+ }
+ return row -> {
+ Object[] fields = new Object[valueType.getTotalFields()];
+ for (int i = 0; i < valueIndex.length; i++) {
+ fields[i] = row.getField(valueIndex[i]);
+ }
+ return new SeaTunnelRow(fields);
Review Comment:
You need to set Rowkind and TableId
```
SeaTunnelRow newRow = new SeaTunnelRow(fields);
newRow.setRowKind(row.getRowKind());
newRow.setTableId(row.getTableId());
return newRow;
```
##########
seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/serialize/DefaultSeaTunnelRowSerializer.java:
##########
@@ -288,6 +378,21 @@ private static Function<SeaTunnelRow, SeaTunnelRow> createKeyRowExtractor(
};
}
+ private static Function<SeaTunnelRow, SeaTunnelRow> createValueRowExtractor(
+ SeaTunnelRowType valueType, List<String> headerFieldNames, SeaTunnelRowType rowType) {
+ int[] valueIndex = new int[valueType.getTotalFields()];
+ for (int i = 0; i < valueType.getTotalFields(); i++) {
+ valueIndex[i] = rowType.indexOf(valueType.getFieldName(i));
+ }
+ return row -> {
+ Object[] fields = new Object[valueType.getTotalFields()];
+ for (int i = 0; i < valueIndex.length; i++) {
+ fields[i] = row.getField(valueIndex[i]);
+ }
+ return new SeaTunnelRow(fields);
Review Comment:
You need to set Rowkind and TableId
```
SeaTunnelRow newRow = new SeaTunnelRow(fields);
newRow.setRowKind(row.getRowKind());
newRow.setTableId(row.getTableId());
return newRow;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "corgy-w (via GitHub)" <gi...@apache.org>.
corgy-w merged PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on code in PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#discussion_r2772923108
##########
seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/sink/KafkaSinkWriter.java:
##########
@@ -260,6 +290,26 @@ private List<String> getPartitionKeyFields(
return Collections.emptyList();
}
+ private List<String> getHeaderFields(
+ ReadonlyConfig pluginConfig, SeaTunnelRowType seaTunnelRowType) {
+
+ if (pluginConfig.get(KAFKA_HEADERS_FIELDS) != null) {
+ List<String> headerFields = pluginConfig.get(KAFKA_HEADERS_FIELDS);
+ List<String> rowTypeFieldNames = Arrays.asList(seaTunnelRowType.getFieldNames());
+ for (String headerField : headerFields) {
+ if (!rowTypeFieldNames.contains(headerField)) {
+ throw new KafkaConnectorException(
+ CommonErrorCodeDeprecated.ILLEGAL_ARGUMENT,
+ String.format(
+ "Header field not found: %s, rowType: %s",
+ headerField, rowTypeFieldNames));
Review Comment:
ditto
##########
seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/serialize/DefaultSeaTunnelRowSerializer.java:
##########
@@ -182,6 +204,32 @@ private static Function<SeaTunnelRow, Iterable<Header>> headersExtractor(
convertToKafkaHeaders((Map<String, String>) row.getField(rowType.indexOf(HEADERS)));
}
+ private static Function<SeaTunnelRow, Iterable<Header>> headersExtractor(
+ List<String> headerFields, SeaTunnelRowType rowType) {
+ if (headerFields == null || headerFields.isEmpty()) {
+ return row -> null;
+ }
+
+ int[] headerFieldIndexes = new int[headerFields.size()];
+ for (int i = 0; i < headerFields.size(); i++) {
+ headerFieldIndexes[i] = rowType.indexOf(headerFields.get(i));
+ }
+
+ return row -> {
+ RecordHeaders kafkaHeaders = new RecordHeaders();
+ for (int i = 0; i < headerFields.size(); i++) {
+ String headerName = headerFields.get(i);
+ Object headerValue = row.getField(headerFieldIndexes[i]);
+ // Write "null" string for null values to keep fields in headers
+ // (consistent with partition_key_fields behavior)
+ String valueStr = headerValue != null ? headerValue.toString() : "null";
Review Comment:
Converting `null` to a literal `"null"` string distorts the actual data and makes it indistinguishable from a real string `"null"`. Please consider using a null payload in the header.
##########
seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/serialize/DefaultSeaTunnelRowSerializer.java:
##########
@@ -182,6 +204,32 @@ private static Function<SeaTunnelRow, Iterable<Header>> headersExtractor(
convertToKafkaHeaders((Map<String, String>) row.getField(rowType.indexOf(HEADERS)));
}
+ private static Function<SeaTunnelRow, Iterable<Header>> headersExtractor(
+ List<String> headerFields, SeaTunnelRowType rowType) {
+ if (headerFields == null || headerFields.isEmpty()) {
+ return row -> null;
+ }
+
+ int[] headerFieldIndexes = new int[headerFields.size()];
+ for (int i = 0; i < headerFields.size(); i++) {
+ headerFieldIndexes[i] = rowType.indexOf(headerFields.get(i));
+ }
+
+ return row -> {
+ RecordHeaders kafkaHeaders = new RecordHeaders();
+ for (int i = 0; i < headerFields.size(); i++) {
+ String headerName = headerFields.get(i);
+ Object headerValue = row.getField(headerFieldIndexes[i]);
+ // Write "null" string for null values to keep fields in headers
+ // (consistent with partition_key_fields behavior)
+ String valueStr = headerValue != null ? headerValue.toString() : "null";
+ kafkaHeaders.add(
+ new RecordHeader(headerName, valueStr.getBytes(StandardCharsets.UTF_8)));
+ }
+ return kafkaHeaders.toArray().length > 0 ? kafkaHeaders : null;
Review Comment:
```suggestion
return kafkaHeaders.iterator().hasNext() ? kafkaHeaders : null;
```
##########
seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/sink/KafkaSinkWriter.java:
##########
@@ -199,10 +206,26 @@ private SeaTunnelRowSerializer<byte[], byte[]> getSerializer(
"Cannot select both `partiton` and `partition_key_fields`. You can configure only one of them");
}
+ // Validate that partition_key_fields and kafka_headers_fields don't overlap
+ List<String> partitionKeyFields = getPartitionKeyFields(pluginConfig, seaTunnelRowType);
+ List<String> headerFields = getHeaderFields(pluginConfig, seaTunnelRowType);
+ if (!partitionKeyFields.isEmpty() && !headerFields.isEmpty()) {
+ for (String headerField : headerFields) {
+ if (partitionKeyFields.contains(headerField)) {
+ throw new KafkaConnectorException(
+ CommonErrorCodeDeprecated.ILLEGAL_ARGUMENT,
+ String.format(
+ "Field '%s' cannot be in both partition_key_fields and kafka_headers_fields",
+ headerField));
Review Comment:
Let's use https://github.com/apache/seatunnel/blob/4d12727ad9325f2fa2245cc0d97f13e39e348393/seatunnel-common/src/main/java/org/apache/seatunnel/common/exception/CommonError.java#L300-L305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "thehkkim (via GitHub)" <gi...@apache.org>.
thehkkim commented on PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#issuecomment-3757778053
@zhangshenghang
Thank you for the detailed review! I've fixed all the critical issues:
**[P0] Header fields with null values are permanently lost (COR-001)**
- ✅ Fixed by writing "null" string instead of skipping null values
- This is now consistent with partition_key_fields behavior
**[MAJOR] No overlap validation between partition_key_fields and kafka_headers_fields (GEN-001)**
- ✅ Added validation to reject overlapping fields
- Implemented check in KafkaSinkWriter.java that throws exception if same field appears in both configs
**[MINOR] NATIVE format silently ignores kafka_headers_fields (GEN-002)**
- ✅ Added validation to reject kafka_headers_fields config for NATIVE format
- Now throws clear error message: "kafka_headers_fields is not supported with NATIVE format"
- Users will see error instead of silent failure
All changes have been tested. Ready for re-review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "thehkkim (via GitHub)" <gi...@apache.org>.
thehkkim commented on PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#issuecomment-3857524438
@dybyte
Just following up on the e2e test implementation I added. The test covers the kafka_headers_fields functionality as expected. Looking forward to your review!
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "thehkkim (via GitHub)" <gi...@apache.org>.
thehkkim commented on PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#issuecomment-3796653918
@och5351
Done! I've squashed all commits into a single commit with the title
matching the PR title: `[Feature][Connector-V2][Kafka] Add support for Kafka message header`
@dybyte
Good suggestion! I've added an e2e test in KafkaIT to verify the headers functionality.
The test (`testSinkKafkaWithHeaders`) verifies:
- Fields specified in `kafka_headers_fields` are correctly sent as Kafka message headers
- These fields are excluded from the message payload
- Non-header fields remain in the payload as expected
Test file: `seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e/src/test/java/org/apache/seatunnel/e2e/connector/kafka/KafkaIT.java:299-336`
Config: `seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e/src/test/resources/kafka_sink_with_headers.conf`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "thehkkim (via GitHub)" <gi...@apache.org>.
thehkkim commented on PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#issuecomment-3882527288
@davidzollo @zhangshenghang
Could you please review this PR?
I'd be grateful for any feedback or suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#issuecomment-3755044596
<!-- code-pr-reviewer -->
Thanks for the contribution! However, there are critical issues that must be addressed before merge.
**[P0] Header fields with null values are permanently lost (COR-001)**
- `DefaultSeaTunnelRowSerializer.java:223` skips null values when writing headers
- `DefaultSeaTunnelRowSerializer.java:350-355` unconditionally excludes all headerFields from value regardless of null values
- Impact: Fields with null values disappear from both headers and value with no warning. Consider writing "null" string (consistent with partition_key_fields behavior), or retaining null fields in value.
**[MAJOR] No overlap validation between partition_key_fields and kafka_headers_fields (GEN-001)**
- `KafkaSinkWriter.java:252-290` validates both configs independently without checking for intersection
- Impact: Same field can appear in both key and headers, then be excluded from value, causing data loss and confusion. Add validation to reject overlapping fields (similar to existing L197-201 pattern).
**[MINOR] NATIVE format silently ignores kafka_headers_fields (GEN-002)**
- `KafkaSinkWriter.java:187-190` calls a create method that doesn't support headerFields parameter
- Impact: Configuration takes effect with no error but headers are not written. Either add validation to reject the config for NATIVE format, or document this limitation clearly.
Please fix at least the P0 and MAJOR issues before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on code in PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#discussion_r2778931598
##########
docs/en/connectors/sink/Kafka.md:
##########
@@ -90,6 +91,23 @@ If not set partition key fields, the null message key will be sent to.
The format of the message key is json, If name is set as the key, for example '{"name":"Jack"}'.
The selected field must be an existing field in the upstream.
+### Kafka Headers Fields
+
+For example, if you want to use value of fields from upstream data as kafka message headers, you can assign field names to this property.
+
+Upstream data is the following:
+
+| name | age | data | source | traceId |
+|------|-----|---------------|--------|-----------|
+| Jack | 16 | data-example1 | web | trace-123 |
+| Mary | 23 | data-example2 | mobile | trace-456 |
+
+If source and traceId are set as the kafka headers fields, then these field values will be added as headers to the kafka message.
+For example, the first row will have headers: `source=web` and `traceId=trace-123`.
+The field values will be converted to strings and used as header values.
+If a field value is null, it will not be added to the headers.
Review Comment:
This part of the documentation should be deleted to reflect the changes in how null values are now handled in headers.
##########
docs/en/connectors/sink/Kafka.md:
##########
@@ -90,6 +91,23 @@ If not set partition key fields, the null message key will be sent to.
The format of the message key is json, If name is set as the key, for example '{"name":"Jack"}'.
The selected field must be an existing field in the upstream.
+### Kafka Headers Fields
+
+For example, if you want to use value of fields from upstream data as kafka message headers, you can assign field names to this property.
+
+Upstream data is the following:
+
+| name | age | data | source | traceId |
+|------|-----|---------------|--------|-----------|
+| Jack | 16 | data-example1 | web | trace-123 |
+| Mary | 23 | data-example2 | mobile | trace-456 |
+
+If source and traceId are set as the kafka headers fields, then these field values will be added as headers to the kafka message.
+For example, the first row will have headers: `source=web` and `traceId=trace-123`.
Review Comment:
Could you please add a note in the documentation explaining that these header fields will be excluded from the message value?
##########
docs/zh/connectors/sink/Kafka.md:
##########
@@ -89,6 +90,23 @@ NON 不提供任何保证:如果 Kafka 代理出现问题,消息可能会丢
消息 key 的格式为 json,如果设置 name 为 key,例如 `{"name":"Jack"}`。
所选的字段必须是上游数据中已存在的字段。
+### Kafka Headers 字段
+
+例如,如果你想使用上游数据中的字段值作为 kafka 消息的 headers,可以将这些字段名指定给此属性。
+
+上游数据如下所示:
+
+| name | age | data | source | traceId |
+|------|-----|---------------|--------|-----------|
+| Jack | 16 | data-example1 | web | trace-123 |
+| Mary | 23 | data-example2 | mobile | trace-456 |
+
+如果将 source 和 traceId 设置为 kafka headers 字段,那么这些字段值将作为 headers 添加到 kafka 消息中。
+例如,第一行将具有 headers:`source=web` 和 `traceId=trace-123`。
+字段值将被转换为字符串并用作 header 值。
+如果字段值为 null,则不会添加到 headers 中。
+所选的字段必须是上游数据中已存在的字段。
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "thehkkim (via GitHub)" <gi...@apache.org>.
thehkkim commented on PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#issuecomment-3866227888
@dybyte
Good catch — I missed this part and should have double-checked the changes more carefully.
This has now been updated to use CommonErrorCode as well and committed.
Thanks for the detailed review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "och5351 (via GitHub)" <gi...@apache.org>.
och5351 commented on PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#issuecomment-3785247116
Hi, @thehkkim !
LGTM +1
However, it appears that the commits should be squashed and the commit messages
should align with your PR title. PTAL.
<img width="898" height="476" alt="image" src="https://github.com/user-attachments/assets/2d7b98ed-0300-4e22-a482-0c2c0aaf97e9" />
I hope this feature can be merged, as it would be beneficial for our requirements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "thehkkim (via GitHub)" <gi...@apache.org>.
thehkkim commented on PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#issuecomment-3864388274
@dybyte
Thanks for the review! I’ve updated the code accordingly:
1. Replaced CommonErrorCodeDeprecated with CommonErrorCode.
2. Followed the suggestion to keep header values as null instead of converting them to the literal "null" string, since converting null to "null" distorts the actual data and makes it indistinguishable from a real "null" value.
3. Updated the empty-check logic from
`return kafkaHeaders.toArray().length > 0 ? kafkaHeaders : null;`
to
`return kafkaHeaders.iterator().hasNext() ? kafkaHeaders : null;`
These changes have been committed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on code in PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#discussion_r2777500407
##########
seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/sink/KafkaSinkWriter.java:
##########
@@ -184,6 +185,12 @@ private SeaTunnelRowSerializer<byte[], byte[]> getSerializer(
MessageFormat messageFormat = pluginConfig.get(FORMAT);
String topic = pluginConfig.get(TOPIC);
if (MessageFormat.NATIVE.equals(messageFormat)) {
+ // Validate that kafka_headers_fields is not configured for NATIVE format
+ if (pluginConfig.get(KAFKA_HEADERS_FIELDS) != null) {
+ throw new KafkaConnectorException(
+ CommonErrorCodeDeprecated.ILLEGAL_ARGUMENT,
Review Comment:
```suggestion
CommonErrorCode.OPERATION_NOT_SUPPORTED,
```
I missed this part earlier. It would be great if we could also use `CommonErrorCode` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][Kafka] Add support for Kafka message header [seatunnel]
Posted by "thehkkim (via GitHub)" <gi...@apache.org>.
thehkkim commented on PR #10335:
URL: https://github.com/apache/seatunnel/pull/10335#issuecomment-3867170883
@dybyte
Thanks for the suggestion.
I’ve updated the documentation (including the Chinese version) to clarify that fields configured as Kafka headers are excluded from the message value (payload).
I’ve also removed the statement saying that null field values are not added to the headers, to reflect the updated header handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org