You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "BK202503 (via GitHub)" <gi...@apache.org> on 2026/06/10 16:26:27 UTC
[PR] KAFKA-20656: Struct honors ByteBuffer remaining bytes [kafka]
BK202503 opened a new pull request, #22536:
URL: https://github.com/apache/kafka/pull/22536
JIRA: [KAFKA-20656](https://issues.apache.org/jira/browse/KAFKA-20656)
### What
Kafka Connect BYTES values may be either `byte[]` or `ByteBuffer`, and `ConnectSchema` recommends `ByteBuffer` because plain arrays do not implement content-based `equals()`/`hashCode()`. `Struct` did not honor that contract:
- `getBytes(...)` called `ByteBuffer.array()`, returning the entire backing array instead of the logical remaining bytes, and throwing `UnsupportedOperationException` for direct buffers.
- `equals` and `hashCode` delegated straight to `Arrays.deepEquals`/`deepHashCode` over the raw `values` array, so two structs holding the same logical BYTES value supplied as `byte[]` vs. `ByteBuffer` were not equal.
### Changes
- `Struct.getBytes(String)` now goes through `Utils.toArray(ByteBuffer)`. That copies the buffer's remaining bytes and works for direct buffers — the same approach `Values.convertToBytes` already uses elsewhere in this module.
- `Struct.equals(Object)` and `Struct.hashCode()` now compare/hash a `normalizedBytesValues()` view of the struct that copies top-level BYTES fields stored as `ByteBuffer` into `byte[]`. The underlying `values` array is not mutated, so `get(String)` callers still see whatever representation was put in. Non-BYTES fields go through `Arrays.deepEquals`/`deepHashCode` unchanged.
The normalization is scoped to top-level BYTES fields, matching the reporter's reproducer. Nested BYTES inside `ARRAY`/`MAP`/`STRUCT` fields keep the previous behavior in this PR.
### Tests
Added three regression tests in `StructTest` that fail against the previous implementation and pass with this change:
- `testGetBytesPreservesByteBufferRemainingBytes`: a sliced `ByteBuffer` returns only the logical bytes.
- `testGetBytesSupportsDirectByteBuffer`: a direct buffer serializes instead of throwing.
- `testEqualsAndHashCodeWithEquivalentByteArrayAndByteBufferValues`: a `byte[]`-valued struct and a `ByteBuffer`-valued struct with the same logical content are `.equals` and share a `hashCode`.
### Validation
```
./gradlew :connect:api:test --tests "org.apache.kafka.connect.data.StructTest.*"
```
All `StructTest` tests pass on JDK 17, including the new three, the existing `testFlatStruct`, and the existing `testEqualsAndHashCodeWithByteArrayValue` (which exercises the unchanged `byte[]`-only equality path).
### Scope
This PR completes the four-ticket `ByteBuffer.array()` cluster for Kafka Connect BYTES values; the other three are split into independent PRs to keep each reviewable:
- KAFKA-20657 (`JsonConverter`) — #22533
- KAFKA-20658 (`Cast` SMT) — #22534
- KAFKA-20666 (offset backing stores) — #22535
### Committer Checklist
- [x] Verified design and implementation
- [x] Verified test coverage and CI build status
- [x] Verified documentation (including upgrade notes) updates (no public API surface change beyond bug-fix behavior; equality semantics now match the documented BYTES contract)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20656: Struct honors ByteBuffer remaining bytes [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22536:
URL: https://github.com/apache/kafka/pull/22536#issuecomment-5041750524
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20656: Struct honors ByteBuffer remaining bytes [kafka]
Posted by "BK202503 (via GitHub)" <gi...@apache.org>.
BK202503 commented on PR #22536:
URL: https://github.com/apache/kafka/pull/22536#issuecomment-5207215941
> I left a comment on Jira. Please take a look.
https://issues.apache.org/jira/browse/KAFKA-20656
(I have no account above link to add comment, so write on the comment below.)
@chia7712
Real producers of ByteBuffer-valued BYTES Structs:
* debezium JdbcValueConverters.java (lines 916, 919, 922): every BLOB/BYTEA/BINARY/RAW column is wrapped in a ByteBuffer before being written via Struct.put(...).
* debezium LogicalDecodingMessageMonitor.java (lines 95, 125, 127): logical decoding messages are wrapped with ByteBuffer.wrap(...) and emitted as BYTES.
* confluentinc/kafka-connect-jdbc GenericDatabaseDialect.java (lines 1911–1922, 2159–2172): the sink already assumes BYTES values may be ByteBuffers and safely handles them by slicing before reading.
* Aiven-Open/jdbc-connector-for-apache-kafka carries the same implementation in its fork.
* mongo-kafka BytesFieldConverter: similarly assumes BYTES may be a ByteBuffer, although it currently calls .array(), which has the same issue on the sink side.
Consumers are already working around the Struct.getBytes(...) limitation. For example, debezium S3LargeMessagePostProcessor.java (lines 289–297) uses duplicate(), remaining(), and get(byte[]) instead of calling .array() directly.
There is also precedent within Kafka itself. KAFKA-16844 (PR #16101) updated ByteArrayConverter to accept ByteBuffer, following the same compatibility principle.
Regarding allocateDirect(...) inside connectors, I couldn't find any evidence of connectors producing direct ByteBuffers. We can therefore drop the direct-buffer test if desired and keep only:
* sliced ByteBuffer test coverage, and
* byte[]/ByteBuffer behavioral equivalence tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20656: Struct honors ByteBuffer remaining bytes [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #22536:
URL: https://github.com/apache/kafka/pull/22536#issuecomment-5161654492
I left a comment on Jira. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20656: Struct honors ByteBuffer remaining bytes [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22536:
URL: https://github.com/apache/kafka/pull/22536#issuecomment-5612154060
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20656: Struct honors ByteBuffer remaining bytes [kafka]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22536:
URL: https://github.com/apache/kafka/pull/22536#issuecomment-4738063717
A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the `triage` label
should be removed to prevent this automation from happening again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20656: Struct honors ByteBuffer remaining bytes [kafka]
Posted by "BK202503 (via GitHub)" <gi...@apache.org>.
BK202503 commented on PR #22536:
URL: https://github.com/apache/kafka/pull/22536#issuecomment-5035595095
@bbejeck, @chia7712 any update for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org