[jira] [Commented] (KAFKA-4852) ByteBufferSerializer not compatible with offsets

["A. Sophie Blee-Goldman (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-4852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17647824#comment-17647824 ] 

A. Sophie Blee-Goldman commented on KAFKA-4852:
-----------------------------------------------

[~LSK] [~wdaehn] is there a reason this ticket was not resolved, or was that just an oversight?

I'm going to close it since the PR seems to have been merged, if there was some newly found issue please reopen this or file a new issue if applicable

> ByteBufferSerializer not compatible with offsets
> ------------------------------------------------
>
>                 Key: KAFKA-4852
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4852
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients
>    Affects Versions: 0.10.1.1
>         Environment: all
>            Reporter: Werner Daehn
>            Assignee: LinShunkang
>            Priority: Minor
>             Fix For: 3.4.0
>
>
> Quick intro: A ByteBuffer.rewind() resets the position to zero. What if the ByteBuffer was created with an offset? new ByteBuffer(data, 3, 10)? The ByteBufferSerializer will send from pos=0 and not from pos=3 onwards.
> Solution: No rewind() but flip() for reading a ByteBuffer. That's what the flip is meant for.
> Story:
> Imagine the incoming data comes from a byte[], e.g. a network stream containing topicname, partition, key, value, ... and you want to create a new ProducerRecord for that. As the constructor of ProducerRecord requires (topic, partition, key, value) you have to copy from above byte[] the key and value. That means there is a memcopy taking place. Since the payload can be potentially large, that introduces a lot of overhead. Twice the memory.
> A nice solution to this problem is to simply wrap the network byte[] into new ByteBuffers:
> ByteBuffer key = ByteBuffer.wrap(data, keystart, keylength);
> ByteBuffer value = ByteBuffer.wrap(data, valuestart, valuelength);
> and then use the ByteBufferSerializer instead of the ByteArraySerializer.
> But that does not work as the ByteBufferSerializer does a rewind(), hence both, key and value, will start at position=0 of the data[].
> public class ByteBufferSerializer implements Serializer<ByteBuffer> {
>     public byte[] serialize(String topic, ByteBuffer data) {
>          data.rewind();



--
This message was sent by Atlassian Jira
(v8.20.10#820010)