You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "efcasado (via GitHub)" <gi...@apache.org> on 2026/03/15 19:36:35 UTC
[PR] [fix][io][kca] kafka headers silently dropped [pulsar]
efcasado opened a new pull request, #25325:
URL: https://github.com/apache/pulsar/pull/25325
### Motivation
The current implementation of the Kafka Connect adaptor silently discards headers from the Kafka Connect SourceRecord instead of mapping them to Pulsar message properties. This is particularly impactful when using [Debezium's Outbox Event Router SMT](https://debezium.io/documentation//reference/3.5/transformations/outbox-event-router.html) with the [table.fields.additional.placement](https://debezium.io/documentation//reference/3.5/transformations/outbox-event-router.html#emitting-messages-with-additional-fields) option, which allows outbox table columns to be placed in the `header`, `envelope`, or `partition` of the resulting record. Columns configured with header placement are silently ignored because the adaptor never propagates Connect record headers to Pulsar message properties. Since Kafka Connect headers and Pulsar message properties are semantically equivalent (string key/value metadata outside the payload), the expected behavior is a direct mapping.
A similar issue was previously identified for `pulsar-io-kafka` and addressed in [#17829](https://github.com/apache/pulsar/pull/17829). However, `kafka-connect-adaptor` was not covered by that fix and still silently discards Kafka Connect SourceRecord headers.
### Modifications
`AbstractKafkaConnectSource` previously returned a hard-coded empty map from `getProperties()`, causing all Kafka Connect headers to be silently dropped. This change maps headers from the transformed `SourceRecord` to Pulsar message properties, preserving metadata that would otherwise be lost in transit (eg. columns placed in header position via Debezium's Outbox Event Router SMT and `table.fields.additional.placement`).
### Verifying this change
- [x] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- Extended the existing `KafkaConnectSourceTest` test suite with an additional test to confirm Kafka headers are correctly propagated to the resulting record.
### Documentation
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: https://github.com/efcasado/pulsar/pull/2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][io][kca] kafka headers silently dropped [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #25325:
URL: https://github.com/apache/pulsar/pull/25325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][io][kca] kafka headers silently dropped [pulsar]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25325:
URL: https://github.com/apache/pulsar/pull/25325#issuecomment-4065801953
## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/25325?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `66.66667%` with `4 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 72.75%. Comparing base ([`735b429`](https://app.codecov.io/gh/apache/pulsar/commit/735b429ca3393b840fe11f97788d3133608e9f02?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`1ce5781`](https://app.codecov.io/gh/apache/pulsar/commit/1ce5781a996482db5c98b44b5e18128bb9cd0283?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 3 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/25325?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...he/pulsar/io/kafka/connect/KafkaConnectSource.java](https://app.codecov.io/gh/apache/pulsar/pull/25325?src=pr&el=tree&filepath=pulsar-io%2Fkafka-connect-adaptor%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fio%2Fkafka%2Fconnect%2FKafkaConnectSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWlvL2thZmthLWNvbm5lY3QtYWRhcHRvci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2lvL2thZmthL2Nvbm5lY3QvS2Fma2FDb25uZWN0U291cmNlLmphdmE=) | 60.00% | [1 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/25325?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
[](https://app.codecov.io/gh/apache/pulsar/pull/25325?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
```diff
@@ Coverage Diff @@
## master #25325 +/- ##
============================================
- Coverage 72.97% 72.75% -0.22%
+ Complexity 34347 34232 -115
============================================
Files 1954 1954
Lines 154715 154725 +10
Branches 17704 17708 +4
============================================
- Hits 112898 112575 -323
- Misses 32816 33072 +256
- Partials 9001 9078 +77
```
| [Flag](https://app.codecov.io/gh/apache/pulsar/pull/25325/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [inttests](https://app.codecov.io/gh/apache/pulsar/pull/25325/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `25.74% <ø> (-0.49%)` | :arrow_down: |
| [systests](https://app.codecov.io/gh/apache/pulsar/pull/25325/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `22.56% <ø> (-0.06%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pulsar/pull/25325/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.73% <66.66%> (-0.22%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/25325?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...r/io/kafka/connect/AbstractKafkaConnectSource.java](https://app.codecov.io/gh/apache/pulsar/pull/25325?src=pr&el=tree&filepath=pulsar-io%2Fkafka-connect-adaptor%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fio%2Fkafka%2Fconnect%2FAbstractKafkaConnectSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWlvL2thZmthLWNvbm5lY3QtYWRhcHRvci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2lvL2thZmthL2Nvbm5lY3QvQWJzdHJhY3RLYWZrYUNvbm5lY3RTb3VyY2UuamF2YQ==) | `69.73% <100.00%> (+0.65%)` | :arrow_up: |
| [...he/pulsar/io/kafka/connect/KafkaConnectSource.java](https://app.codecov.io/gh/apache/pulsar/pull/25325?src=pr&el=tree&filepath=pulsar-io%2Fkafka-connect-adaptor%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fio%2Fkafka%2Fconnect%2FKafkaConnectSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWlvL2thZmthLWNvbm5lY3QtYWRhcHRvci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2lvL2thZmthL2Nvbm5lY3QvS2Fma2FDb25uZWN0U291cmNlLmphdmE=) | `70.00% <60.00%> (-0.67%)` | :arrow_down: |
... and [89 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/25325/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org