You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "LinShunkang (Jira)" <ji...@apache.org> on 2022/09/25 07:26:00 UTC
[jira] [Commented] (KAFKA-4852) ByteBufferSerializer not compatible with offsets
[ https://issues.apache.org/jira/browse/KAFKA-4852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609089#comment-17609089 ]
LinShunkang commented on KAFKA-4852:
------------------------------------
Agree! I also found this bug:
{code:java}
@Test
public void testByteBufferSerializer() {
final byte[] bytes = "Hello".getBytes(UTF_8);
final ByteBuffer buffer = ByteBuffer.allocate(7);
buffer.put(bytes);
try (final ByteBufferSerializer serializer = new ByteBufferSerializer()) {
assertArrayEquals(bytes, serializer.serialize(topic, buffer));
}
} {code}
Executing the above test case will throw the following exception:
{code:java}
array lengths differ, expected: <5> but was: <7>
Expected :5
Actual :7
<Click to see difference>org.opentest4j.AssertionFailedError: array lengths differ, expected: <5> but was: <7>
...
at org.apache.kafka.common.serialization.SerializationTest.testByteBufferSerializer(SerializationTest.java:397)
...
at java.util.ArrayList.forEach(ArrayList.java:1259)
...
at java.util.ArrayList.forEach(ArrayList.java:1259)
...
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74){code}
> ByteBufferSerializer not compatible with offsets
> ------------------------------------------------
>
> Key: KAFKA-4852
> URL: https://issues.apache.org/jira/browse/KAFKA-4852
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Affects Versions: 0.10.1.1
> Environment: all
> Reporter: Werner Daehn
> Priority: Minor
>
> Quick intro: A ByteBuffer.rewind() resets the position to zero. What if the ByteBuffer was created with an offset? new ByteBuffer(data, 3, 10)? The ByteBufferSerializer will send from pos=0 and not from pos=3 onwards.
> Solution: No rewind() but flip() for reading a ByteBuffer. That's what the flip is meant for.
> Story:
> Imagine the incoming data comes from a byte[], e.g. a network stream containing topicname, partition, key, value, ... and you want to create a new ProducerRecord for that. As the constructor of ProducerRecord requires (topic, partition, key, value) you have to copy from above byte[] the key and value. That means there is a memcopy taking place. Since the payload can be potentially large, that introduces a lot of overhead. Twice the memory.
> A nice solution to this problem is to simply wrap the network byte[] into new ByteBuffers:
> ByteBuffer key = ByteBuffer.wrap(data, keystart, keylength);
> ByteBuffer value = ByteBuffer.wrap(data, valuestart, valuelength);
> and then use the ByteBufferSerializer instead of the ByteArraySerializer.
> But that does not work as the ByteBufferSerializer does a rewind(), hence both, key and value, will start at position=0 of the data[].
> public class ByteBufferSerializer implements Serializer<ByteBuffer> {
> public byte[] serialize(String topic, ByteBuffer data) {
> data.rewind();
--
This message was sent by Atlassian Jira
(v8.20.10#820010)